2024-03-25T16:07:03.891 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:07:03.892 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/orig.config.yaml 2024-03-25T16:07:03.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:07:04.030 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621247' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIgV9nVxjA/tkjSmI2/rMBJUsU3ZgOjTL1q0uxeQIB2jJz7E19yBi0JwhUL3MOe4vvRZsUDnne1+evrxO55qos0= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBt6g2fh+YzZnvAAW6vAoxLl/uWoK/Ld3GV5Mvn9yC9qpXLYU65zgeGJ1QY6FY0NWTToyw6y3UDkkKKlSlz75Ec= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:07:04.030 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:07:04.031 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:07:04.031 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:07:04.033 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:07:04.058 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:07:04.058 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:07:04.058 INFO:teuthology.packaging:ref: None 2024-03-25T16:07:04.058 INFO:teuthology.packaging:tag: None 2024-03-25T16:07:04.058 INFO:teuthology.packaging:branch: main 2024-03-25T16:07:04.058 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:07:04.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:07:04.293 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:07:04.294 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:07:04.300 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:07:04.300 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:07:04.308 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:07:04.324 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:07:04.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:07:04.358 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:01:51.987156', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIgV9nVxjA/tkjSmI2/rMBJUsU3ZgOjTL1q0uxeQIB2jJz7E19yBi0JwhUL3MOe4vvRZsUDnne1+evrxO55qos0='} 2024-03-25T16:07:04.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:01:51.989150', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBt6g2fh+YzZnvAAW6vAoxLl/uWoK/Ld3GV5Mvn9yC9qpXLYU65zgeGJ1QY6FY0NWTToyw6y3UDkkKKlSlz75Ec='} 2024-03-25T16:07:04.380 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:07:04.386 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:07:04.386 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:07:04.386 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:07:04.456 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45b9f240d0>, signals=[15]) 2024-03-25T16:07:04.456 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:07:04.463 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:07:04.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T16:07:04.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:07:04.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T16:07:04.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:07:04.653 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:07:04.658 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-03-25T16:07:04.675 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-03-25T16:07:04.675 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:07:04.732 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:07:04.733 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:07:04.733 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-03-25T16:07:04.758 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-03-25T16:07:04.775 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-03-25T16:07:04.776 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-03-25T16:07:04.832 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-03-25T16:07:04.832 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="9" 2024-03-25T16:07:04.832 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-03-25T16:07:04.832 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:07:04.832 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="9" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:07:04.833 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:07:04.833 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-03-25T16:07:04.858 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:07:04.868 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:07:04.988 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:07:04.989 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:07:04.991 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:07:05.006 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:07:05.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:07:05.014 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:07:05.048 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:07:05.158 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:07:05.209 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:07:05.225 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:07:05.225 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:07:05.225 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:07:05.226 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:07:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:07:05.242 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-25T16:07:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:07:05.243 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-03-25T16:07:05.298 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:07:05.299 INFO:teuthology.task.kernel:Running kernel on smithi171: 5.14.0-361.el9.x86_64 2024-03-25T16:07:05.299 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2024-03-25T16:07:05.301 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:07:05.301 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-361.el9.x86_64 2024-03-25T16:07:05.301 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-03-25T16:07:07.444 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:07:07.688 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:07:09.070 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:07:09.372 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:07:11.502 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-25T16:07:13.487 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-25T16:07:18.746 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-25T16:07:18.977 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 893 kB/s | 8.0 MB 00:09 2024-03-25T16:07:20.414 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T16:07:22.198 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:07:23.361 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-25T16:07:27.006 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:07:27.295 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:07:28.654 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-25T16:07:29.104 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:07:29.104 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:07:29.212 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:07:29.224 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:07:29.224 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-03-25T16:07:29.224 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:07:29.224 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:07:29.224 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-03-25T16:07:29.225 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:07:29.226 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 76 M 2024-03-25T16:07:29.226 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 123 M 2024-03-25T16:07:29.226 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:07:29.424 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 284 kB/s | 44 kB 00:00 2024-03-25T16:07:31.341 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-25T16:07:33.741 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 3.3 MB/s | 15 MB 00:04 2024-03-25T16:07:35.416 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 4.6 MB/s | 27 MB 00:05 2024-03-25T16:07:36.213 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:07:36.291 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 4.8 MB/s | 33 MB 00:07 2024-03-25T16:07:36.291 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:07:36.291 INFO:teuthology.orchestra.run.smithi087.stdout:Total 11 MB/s | 76 MB 00:07 2024-03-25T16:07:36.952 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:07:37.333 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:07:37.333 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:07:38.350 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:07:38.351 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:07:38.362 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:07:38.362 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:07:38.476 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:07:38.486 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout:Install 4 Packages 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:07:38.487 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 76 M 2024-03-25T16:07:38.488 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 123 M 2024-03-25T16:07:38.488 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:07:38.590 INFO:teuthology.orchestra.run.smithi171.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 729 kB/s | 44 kB 00:00 2024-03-25T16:07:39.724 INFO:teuthology.orchestra.run.smithi171.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-25T16:07:40.315 INFO:teuthology.orchestra.run.smithi171.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 19 MB/s | 33 MB 00:01 2024-03-25T16:07:40.591 INFO:teuthology.orchestra.run.smithi171.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 14 MB/s | 27 MB 00:02 2024-03-25T16:07:40.591 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:07:40.591 INFO:teuthology.orchestra.run.smithi171.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-25T16:07:40.832 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:07:41.254 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:07:41.586 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:07:41.619 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:07:41.619 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:07:41.683 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:07:42.610 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:07:42.610 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:07:44.531 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:07:44.670 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:07:45.069 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:07:45.682 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:07:45.768 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:07:48.509 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:07:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:07:49.365 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:07:49.461 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:07:53.386 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:07:53.420 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:07:54.154 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:07:58.180 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:23.277 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:23.277 INFO:teuthology.orchestra.run.smithi087.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:08:23.277 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:08:23.320 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:23.710 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:23.710 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:08:23.710 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:08:23.710 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.096 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.096 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.096 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:08:24.096 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:08:24.429 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:08:24.525 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.526 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:08:24.526 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:08:24.527 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:08:24.527 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:08:24.527 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:08:24.527 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:08:24.527 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:08:24.607 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:24.607 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:08:24.608 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:08:24.641 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:08:24.642 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:08:24.643 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:08:24.643 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-03-25T16:08:24.643 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:08:24.643 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:08:24.644 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:08:24.644 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:08:24.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:08:25.250 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-03-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:08:26.622 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-03-25T16:08:26.625 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-03-25T16:08:26.640 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.ByZM4ikLao 2024-03-25T16:08:26.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ByZM4ikLao 2024-03-25T16:08:26.708 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.ByZM4ikLao 2024-03-25T16:08:26.854 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.ByZM4ikLao is 6KB 2024-03-25T16:08:26.903 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.ByZM4ikLao 2024-03-25T16:08:26.917 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:08:26.983 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:08:26.983 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:08:26.983 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:08:26.983 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:08:26.984 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:08:27.134 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-03-25T16:08:27.875 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:27.876 INFO:teuthology.orchestra.run.smithi171.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:08:27.876 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:08:27.920 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:28.227 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:28.227 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:08:28.228 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:08:28.228 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:08:28.630 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:08:28.631 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:08:28.957 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:08:29.062 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:29.062 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:08:29.062 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:08:29.062 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:08:29.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:08:29.063 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:08:29.063 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:08:29.063 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:08:29.143 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:08:29.143 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:08:29.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.06 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 68.el9 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:08:29.179 INFO:teuthology.orchestra.run.smithi171.stdout:Group : Unspecified 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 8264254 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : builder@centos.org 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:08:29.180 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:08:29.181 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:08:29.182 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:08:29.182 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:08:29.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:08:29.761 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-03-25T16:08:31.208 INFO:teuthology.orchestra.run.smithi171.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:08:31.228 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-03-25T16:08:31.231 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-25T16:08:31.246 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.w2y0IvLMSB 2024-03-25T16:08:31.246 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w2y0IvLMSB 2024-03-25T16:08:31.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.w2y0IvLMSB 2024-03-25T16:08:31.446 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.w2y0IvLMSB is 6KB 2024-03-25T16:08:31.495 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.w2y0IvLMSB 2024-03-25T16:08:31.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:08:31.582 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:08:31.582 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:08:31.582 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:08:31.582 INFO:teuthology.orchestra.run.smithi171.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:08:31.584 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:08:31.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-03-25T16:08:57.158 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:08:57.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:08:57.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:01.742 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:09:01.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:09:01.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:15.551 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-03-25T16:09:20.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-25T16:09:24.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:09:24.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:27.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-03-25T16:09:29.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:09:29.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:32.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-25T16:09:39.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:09:39.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:44.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:09:44.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:02.655 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-03-25T16:10:11.518 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-03-25T16:10:11.987 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:10:11.987 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:10:11.988 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-03-25T16:10:12.001 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:10:12.001 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:10:12.001 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:10:12.002 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:10:13.002 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:10:13.003 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:10:13.130 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-03-25T16:10:13.167 DEBUG:teuthology.parallel:result is None 2024-03-25T16:10:17.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:10:17.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:18.010 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-25T16:10:18.466 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:10:18.466 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:10:18.467 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-03-25T16:10:18.480 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:10:18.481 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:10:18.481 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:10:18.481 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:10:19.482 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:10:19.482 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:10:19.540 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-03-25T16:10:19.570 DEBUG:teuthology.parallel:result is None 2024-03-25T16:10:19.571 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:10:19.579 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:10:19.580 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:10:19.583 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:10:19.616 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:10:19.628 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:10:19.634 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:10:19.635 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:10:19.659 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:10:19.740 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:10:19.748 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:10:19.749 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:10:19.751 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:10:19.782 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:10:19.794 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:10:19.797 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:10:19.812 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:10:19.814 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:10:19.821 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:10:19.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:10:19.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:10:19.866 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:10:19.875 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:10:19.876 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:10:19.893 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:10:19.923 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:10:19.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:10:20.044 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:10:20.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:10:20.171 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T16:10:20.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:10:20.240 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:10:20.280 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:10:20.328 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:10:20.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:10:20.450 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T16:10:20.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:10:20.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-03-25T16:10:20.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-03-25T16:10:20.562 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:10:20.616 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:10:21.106 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:10:21.114 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:10:21.114 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:10:21.134 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:10:21.142 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-03-25T16:10:21.194 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-03-25T16:10:21.196 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-03-25T16:10:21.302 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-03-25T16:10:21.304 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:10:21.304 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-03-25T16:10:21.337 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-03-25T16:10:21.337 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-03-25T16:10:21.390 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-03-25T16:10:21.391 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:10:21.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:10:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:21.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:10:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:21.446 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:10:21.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:10:21.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:10:21.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:10:21.651 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:10:21.659 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:10:21.841 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:10:21.874 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:10:21.900 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:14:58.475 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-03-25T16:14:58.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:14:58.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:58.593 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-03-25T16:14:58.677 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-25T16:14:58.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:14:58.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:58.765 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-03-25T16:14:58.803 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-03-25T16:14:58.803 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:14:58.812 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:14:58.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:14:58.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:14:58.815 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:14:58.816 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:14:58.844 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:14:58.860 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:14:58.877 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:14:58.889 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-03-25T16:14:58.894 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:14:58.902 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-03-25T16:14:58.918 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:14:58.924 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-03-25T16:14:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:14:58.938 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-03-25T16:14:58.954 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:14:58.971 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:14:58.985 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:58.989 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:59.033 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:14:59.035 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:14:59.036 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-03-25T16:14:59.036 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:59.036 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:59.036 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:59.036 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:14:59.037 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:14:59.046 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:14:59.046 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:14:59.046 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:14:59.046 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:14:59.053 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:14:59.053 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:14:59.053 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:14:59.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:14:59.056 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:14:59.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:14:59.277 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:14:59.277 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:14:59.281 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:14:59.281 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:14:59.539 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:14:59.539 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T16:14:59.539 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:14:59.570 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:14:59.570 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:14:59.626 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:14:59.627 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T16:14:59.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:14:59.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:14:59.655 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:14:59.655 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:14:59.722 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:14:59.724 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:14:59.789 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-03-25T16:14:59.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-03-25T16:14:59.805 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:14:59.872 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-03-25T16:14:59.873 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-03-25T16:15:00.027 INFO:teuthology.orchestra.run.smithi087.stdout:74 files removed 2024-03-25T16:15:00.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-03-25T16:15:00.115 INFO:teuthology.orchestra.run.smithi171.stdout:74 files removed 2024-03-25T16:15:00.142 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2024-03-25T16:15:01.043 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 119 kB/s | 82 kB 00:00 2024-03-25T16:15:01.211 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 107 kB/s | 82 kB 00:00 2024-03-25T16:15:01.885 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:15:01.986 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:15:02.586 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-25T16:15:02.778 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T16:15:03.096 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:15:03.339 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:15:04.605 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-25T16:15:04.914 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-25T16:15:06.729 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-25T16:15:07.088 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-25T16:15:08.405 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-25T16:15:12.575 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-25T16:15:14.124 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-03-25T16:15:24.312 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-25T16:15:28.354 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:15:29.997 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T16:15:31.125 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-25T16:15:36.065 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-25T16:15:38.492 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:15:38.499 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:15:38.500 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:15:38.501 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout:Install 30 Packages 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-03-25T16:15:38.502 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:15:38.503 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 58 M 2024-03-25T16:15:38.503 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:15:41.974 INFO:teuthology.orchestra.run.smithi171.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.8 MB/s | 5.3 MB 00:03 2024-03-25T16:15:42.074 INFO:teuthology.orchestra.run.smithi171.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:15:42.458 INFO:teuthology.orchestra.run.smithi171.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.9 MB/s | 742 kB 00:00 2024-03-25T16:15:42.699 INFO:teuthology.orchestra.run.smithi171.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.0 MB/s | 498 kB 00:00 2024-03-25T16:15:44.641 INFO:teuthology.orchestra.run.smithi171.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 2.6 MB/s | 5.1 MB 00:01 2024-03-25T16:15:44.716 INFO:teuthology.orchestra.run.smithi171.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 606 kB/s | 45 kB 00:00 2024-03-25T16:15:44.792 INFO:teuthology.orchestra.run.smithi171.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-25T16:15:45.009 INFO:teuthology.orchestra.run.smithi171.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 3.3 MB/s | 20 MB 00:06 2024-03-25T16:15:45.034 INFO:teuthology.orchestra.run.smithi171.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 669 kB/s | 162 kB 00:00 2024-03-25T16:15:45.151 INFO:teuthology.orchestra.run.smithi171.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.6 MB/s | 9.7 MB 00:06 2024-03-25T16:15:45.201 INFO:teuthology.orchestra.run.smithi171.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-25T16:15:45.226 INFO:teuthology.orchestra.run.smithi171.stdout:(12/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 320 kB 00:00 2024-03-25T16:15:45.310 INFO:teuthology.orchestra.run.smithi171.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 626 kB/s | 99 kB 00:00 2024-03-25T16:15:45.360 INFO:teuthology.orchestra.run.smithi171.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-03-25T16:15:45.386 INFO:teuthology.orchestra.run.smithi171.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T16:15:45.419 INFO:teuthology.orchestra.run.smithi171.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-25T16:15:45.444 INFO:teuthology.orchestra.run.smithi171.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-25T16:15:45.470 INFO:teuthology.orchestra.run.smithi171.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 41 kB 00:00 2024-03-25T16:15:45.503 INFO:teuthology.orchestra.run.smithi171.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T16:15:45.670 INFO:teuthology.orchestra.run.smithi171.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 174 kB/s | 35 kB 00:00 2024-03-25T16:15:45.704 INFO:teuthology.orchestra.run.smithi171.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 222 kB/s | 44 kB 00:00 2024-03-25T16:15:45.779 INFO:teuthology.orchestra.run.smithi171.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 648 kB/s | 216 kB 00:00 2024-03-25T16:15:45.946 INFO:teuthology.orchestra.run.smithi171.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-03-25T16:15:46.088 INFO:teuthology.orchestra.run.smithi171.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 740 kB/s | 309 kB 00:00 2024-03-25T16:15:46.171 INFO:teuthology.orchestra.run.smithi171.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 216 kB/s | 49 kB 00:00 2024-03-25T16:15:46.197 INFO:teuthology.orchestra.run.smithi171.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-25T16:15:46.530 INFO:teuthology.orchestra.run.smithi171.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 575 kB/s | 192 kB 00:00 2024-03-25T16:15:46.614 INFO:teuthology.orchestra.run.smithi171.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-03-25T16:15:46.972 INFO:teuthology.orchestra.run.smithi171.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-03-25T16:15:47.381 INFO:teuthology.orchestra.run.smithi171.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-03-25T16:15:48.489 INFO:teuthology.orchestra.run.smithi171.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 3.4 MB 00:01 2024-03-25T16:15:53.531 INFO:teuthology.orchestra.run.smithi171.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 488 kB/s | 3.1 MB 00:06 2024-03-25T16:15:53.534 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:53.534 INFO:teuthology.orchestra.run.smithi171.stdout:Total 3.8 MB/s | 58 MB 00:15 2024-03-25T16:15:53.568 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:15:53.608 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:15:53.609 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:15:54.088 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:15:54.088 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:15:55.577 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:15:55.672 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:15:55.710 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:15:55.757 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:15:55.784 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:15:55.801 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:15:56.002 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:15:56.054 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:15:56.373 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:15:56.445 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:15:56.520 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:15:56.606 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:15:56.630 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:15:56.676 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:15:56.705 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:15:56.739 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:15:56.807 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:15:56.832 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:15:56.877 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:15:56.905 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:15:57.012 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:15:57.040 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:15:57.081 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:15:57.121 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:15:57.150 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:15:57.183 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:15:57.498 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:15:57.529 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:15:57.601 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:15:57.634 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:15:57.674 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:15:58.129 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:15:58.157 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:15:58.228 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:15:58.297 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:15:58.344 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:16:00.487 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:00.522 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:01.083 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:01.200 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:16:01.284 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:16:01.284 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:16:01.284 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:16:01.284 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:01.326 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:16:10.575 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /root 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-03-25T16:16:10.576 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:11.492 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:16:11.516 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:16:11.516 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:11.516 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:16:11.516 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:16:11.516 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:16:11.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:11.518 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:16:11.595 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:16:11.595 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:16:12.692 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:16:12.692 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:16:12.692 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:16:12.692 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:16:12.692 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:16:12.693 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:16:12.694 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:16:12.695 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:16:12.695 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:16:12.695 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:16:12.695 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:16:12.695 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:16:13.875 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:16:13.876 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.877 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:13.878 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:14.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2024-03-25T16:16:14.766 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:38 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:14.913 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:14.914 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:14.914 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:16:14.914 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:14.914 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:16:14.915 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:14.916 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:16:14.916 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:14.916 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-03-25T16:16:14.916 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:14.917 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 48 M 2024-03-25T16:16:14.917 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 204 M 2024-03-25T16:16:14.917 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:16:15.340 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 596 kB/s | 217 kB 00:00 2024-03-25T16:16:15.557 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-25T16:16:15.607 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T16:16:15.632 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 249 kB/s | 163 kB 00:00 2024-03-25T16:16:15.942 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 561 kB/s | 187 kB 00:00 2024-03-25T16:16:21.144 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 317 kB/s | 21 MB 01:06 2024-03-25T16:16:26.069 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T16:16:26.532 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 4.1 MB/s | 47 MB 00:11 2024-03-25T16:16:26.534 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:26.534 INFO:teuthology.orchestra.run.smithi171.stdout:Total 4.1 MB/s | 48 MB 00:11 2024-03-25T16:16:26.548 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:16:26.567 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:16:26.567 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:16:26.940 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:16:26.940 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:16:27.530 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:16:27.607 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:16:27.660 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:16:27.691 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:16:27.738 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:16:27.762 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:16:28.515 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:16:28.529 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:16:28.529 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:16:28.529 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:16:28.530 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:16:28.531 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:16:28.532 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-03-25T16:16:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:16:28.534 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 58 M 2024-03-25T16:16:28.534 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:16:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:16:32.304 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:16:34.736 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:16:34.736 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:16:34.736 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:16:34.736 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:16:34.737 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:16:34.737 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:16:35.087 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:35.088 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:35.376 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2024-03-25T16:16:35.544 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 9.7 MB 00:06 2024-03-25T16:16:35.644 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:16:35.948 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:59 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:36.104 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:16:36.107 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:16:36.108 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:16:36.109 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout:========================================================================================= 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout:Install 36 Packages 2024-03-25T16:16:36.110 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:36.111 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 30 M 2024-03-25T16:16:36.111 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 107 M 2024-03-25T16:16:36.111 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:16:36.120 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.5 MB/s | 742 kB 00:00 2024-03-25T16:16:36.445 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.5 MB/s | 498 kB 00:00 2024-03-25T16:16:36.858 INFO:teuthology.orchestra.run.smithi171.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T16:16:38.058 INFO:teuthology.orchestra.run.smithi171.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 986 kB/s | 1.5 MB 00:01 2024-03-25T16:16:38.942 INFO:teuthology.orchestra.run.smithi171.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 4.7 MB 00:02 2024-03-25T16:16:39.050 INFO:teuthology.orchestra.run.smithi171.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-25T16:16:39.209 INFO:teuthology.orchestra.run.smithi171.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:16:39.259 INFO:teuthology.orchestra.run.smithi171.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T16:16:39.293 INFO:teuthology.orchestra.run.smithi171.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-25T16:16:39.336 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.8 MB/s | 5.1 MB 00:02 2024-03-25T16:16:39.344 INFO:teuthology.orchestra.run.smithi171.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-03-25T16:16:39.386 INFO:teuthology.orchestra.run.smithi171.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 931 kB/s | 39 kB 00:00 2024-03-25T16:16:39.444 INFO:teuthology.orchestra.run.smithi171.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:16:39.445 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 417 kB/s | 45 kB 00:00 2024-03-25T16:16:39.511 INFO:teuthology.orchestra.run.smithi171.stdout:(11/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 724 kB/s | 2.1 MB 00:02 2024-03-25T16:16:39.570 INFO:teuthology.orchestra.run.smithi171.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-03-25T16:16:39.570 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.0 MB/s | 131 kB 00:00 2024-03-25T16:16:39.595 INFO:teuthology.orchestra.run.smithi171.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-25T16:16:39.620 INFO:teuthology.orchestra.run.smithi171.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:16:39.646 INFO:teuthology.orchestra.run.smithi171.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-25T16:16:39.721 INFO:teuthology.orchestra.run.smithi171.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-25T16:16:39.747 INFO:teuthology.orchestra.run.smithi171.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 995 kB/s | 125 kB 00:00 2024-03-25T16:16:39.779 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 777 kB/s | 162 kB 00:00 2024-03-25T16:16:39.914 INFO:teuthology.orchestra.run.smithi171.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 261 kB/s | 43 kB 00:00 2024-03-25T16:16:39.947 INFO:teuthology.orchestra.run.smithi171.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 669 kB/s | 151 kB 00:00 2024-03-25T16:16:40.056 INFO:teuthology.orchestra.run.smithi171.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T16:16:40.082 INFO:teuthology.orchestra.run.smithi171.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T16:16:40.107 INFO:teuthology.orchestra.run.smithi171.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-03-25T16:16:40.132 INFO:teuthology.orchestra.run.smithi171.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-25T16:16:40.158 INFO:teuthology.orchestra.run.smithi171.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-03-25T16:16:40.183 INFO:teuthology.orchestra.run.smithi171.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-25T16:16:40.217 INFO:teuthology.orchestra.run.smithi171.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-03-25T16:16:40.242 INFO:teuthology.orchestra.run.smithi171.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 787 kB/s | 46 kB 00:00 2024-03-25T16:16:40.284 INFO:teuthology.orchestra.run.smithi171.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-25T16:16:40.318 INFO:teuthology.orchestra.run.smithi171.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T16:16:40.343 INFO:teuthology.orchestra.run.smithi171.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-25T16:16:40.386 INFO:teuthology.orchestra.run.smithi171.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T16:16:40.419 INFO:teuthology.orchestra.run.smithi171.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 472 kB/s | 36 kB 00:00 2024-03-25T16:16:40.461 INFO:teuthology.orchestra.run.smithi171.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T16:16:40.512 INFO:teuthology.orchestra.run.smithi171.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-03-25T16:16:40.537 INFO:teuthology.orchestra.run.smithi171.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-03-25T16:16:41.112 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 240 kB/s | 320 kB 00:01 2024-03-25T16:16:41.704 INFO:teuthology.orchestra.run.smithi171.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 4.6 MB/s | 17 MB 00:03 2024-03-25T16:16:41.708 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:41.708 INFO:teuthology.orchestra.run.smithi171.stdout:Total 5.3 MB/s | 30 MB 00:05 2024-03-25T16:16:41.732 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:16:41.773 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:16:41.774 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:16:41.854 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 417 kB/s | 5.3 MB 00:13 2024-03-25T16:16:41.954 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 997 kB/s | 99 kB 00:00 2024-03-25T16:16:42.121 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 635 kB/s | 106 kB 00:00 2024-03-25T16:16:42.146 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 288 kB/s | 298 kB 00:01 2024-03-25T16:16:42.180 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-25T16:16:42.188 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:16:42.188 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:16:42.205 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-25T16:16:42.273 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-25T16:16:42.315 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-25T16:16:42.340 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 620 kB/s | 41 kB 00:00 2024-03-25T16:16:42.440 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 349 kB/s | 35 kB 00:00 2024-03-25T16:16:42.524 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 535 kB/s | 44 kB 00:00 2024-03-25T16:16:42.549 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 925 kB/s | 216 kB 00:00 2024-03-25T16:16:42.791 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T16:16:42.841 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-03-25T16:16:42.917 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-25T16:16:43.000 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-25T16:16:43.017 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-03-25T16:16:43.093 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:16:43.126 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-25T16:16:43.251 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-25T16:16:43.520 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:16:43.582 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:16:43.622 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:16:43.651 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:16:43.720 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:16:43.767 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:16:43.800 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:16:43.890 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:16:43.969 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:16:44.012 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:16:44.083 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:16:44.138 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:16:44.176 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:16:44.225 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:16:44.280 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:16:44.462 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:16:44.516 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:16:44.554 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:16:44.589 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:16:44.619 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:16:44.694 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:16:44.738 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:16:44.771 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:16:44.812 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:16:44.872 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:16:44.908 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 3.4 MB 00:01 2024-03-25T16:16:44.996 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:16:45.047 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:16:45.118 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:16:45.263 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:16:45.390 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:16:45.550 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:16:45.579 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:16:45.604 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:45.735 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 20 MB 00:17 2024-03-25T16:16:47.098 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:16:47.125 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:16:47.126 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:47.126 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:16:47.126 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:16:47.126 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:16:47.126 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:47.566 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:16:47.589 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:47.800 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:16:47.823 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:47.881 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:16:48.893 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 568 kB/s | 3.1 MB 00:05 2024-03-25T16:16:48.896 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:48.897 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 58 MB 00:20 2024-03-25T16:16:48.928 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:16:48.965 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:16:48.966 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:16:49.439 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:16:49.439 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:16:49.740 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:16:49.741 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:16:49.742 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:16:49.742 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:16:49.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:16:49.744 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:16:50.674 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:16:50.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:16:50.675 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:16:50.676 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:50.677 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:50.916 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:16:50.991 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:16:51.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2024-03-25T16:16:51.022 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:16:51.056 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:16:51.079 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:16:51.095 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:16:51.292 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:16:51.343 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:16:51.580 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:51.663 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:16:51.678 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:16:51.730 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:51.731 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:16:51.732 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:51.733 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:16:51.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install cephadm 2024-03-25T16:16:51.809 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:16:51.895 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:16:51.935 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:16:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:16:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:16:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:16:52.139 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:16:52.162 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:16:52.224 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:16:52.252 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:16:52.359 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:52.374 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:16:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:16:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:16:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:16:52.506 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:16:52.507 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:52.508 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 747 k 2024-03-25T16:16:52.508 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 754 k 2024-03-25T16:16:52.508 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:16:52.522 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:16:52.563 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:16:52.893 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:16:52.918 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:16:53.016 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:16:53.039 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:16:53.085 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:16:53.541 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:16:53.571 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:16:53.642 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:16:53.717 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:16:53.748 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:16:53.933 INFO:teuthology.orchestra.run.smithi171.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 525 kB/s | 747 kB 00:01 2024-03-25T16:16:53.933 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:53.933 INFO:teuthology.orchestra.run.smithi171.stdout:Total 524 kB/s | 747 kB 00:01 2024-03-25T16:16:53.934 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:16:53.940 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:16:53.940 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:16:53.946 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:16:53.946 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:16:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:16:54.478 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:16:54.555 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:16:54.925 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:16:55.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:16:55.195 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:55.195 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:16:55.195 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:16:55.195 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:55.196 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:55.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:16:55.934 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:55.945 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:55.978 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:56.101 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:16:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 145 k 2024-03-25T16:16:56.103 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 436 k 2024-03-25T16:16:56.103 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:16:56.525 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:16:56.623 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 279 kB/s | 145 kB 00:00 2024-03-25T16:16:56.624 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:56.624 INFO:teuthology.orchestra.run.smithi171.stdout:Total 278 kB/s | 145 kB 00:00 2024-03-25T16:16:56.624 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:16:56.632 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:16:56.632 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:16:56.684 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:16:56.684 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:16:56.806 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:16:56.818 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:16:56.891 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:16:56.892 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:16:56.892 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:16:56.892 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:16:56.903 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:16:56.924 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:16:56.928 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:16:56.928 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:16:56.928 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:16:56.928 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:57.894 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:16:57.894 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:57.895 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:16:57.895 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:16:57.895 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:57.895 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:58.098 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2024-03-25T16:16:58.671 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:22 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:58.771 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:16:58.822 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:58.823 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:16:58.823 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:16:58.888 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:16:59.450 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:23 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:16:59.600 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:16:59.601 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-03-25T16:16:59.602 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:16:59.603 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 6.9 M 2024-03-25T16:16:59.603 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 26 M 2024-03-25T16:16:59.603 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:00.351 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 63 kB/s | 31 kB 00:00 2024-03-25T16:17:00.377 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T16:17:00.661 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 840 kB/s | 238 kB 00:00 2024-03-25T16:17:00.711 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-03-25T16:17:00.794 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-03-25T16:17:01.070 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-25T16:17:01.420 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-25T16:17:01.546 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:17:01.654 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 283 kB/s | 31 kB 00:00 2024-03-25T16:17:01.771 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.9 MB/s | 2.0 MB 00:00 2024-03-25T16:17:01.796 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-25T16:17:02.063 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 824 kB/s | 1.8 MB 00:02 2024-03-25T16:17:02.067 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:02.068 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-25T16:17:02.087 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:02.105 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:02.105 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:02.282 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:02.282 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:02.794 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:02.873 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:17:02.908 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:17:02.963 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:17:03.002 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:17:03.114 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:17:03.245 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:17:03.340 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:17:03.392 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:17:03.421 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:17:03.469 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:17:03.712 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:17:03.745 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:17:04.567 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:17:04.567 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:17:04.567 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:17:04.567 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:17:04.568 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:17:04.911 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:17:04.912 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:17:04.912 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:17:04.912 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:17:04.912 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:04.912 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:05.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:17:05.710 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:05.861 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:17:05.862 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-03-25T16:17:05.863 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:05.864 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 37 M 2024-03-25T16:17:05.864 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 209 M 2024-03-25T16:17:05.864 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:06.132 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:17:06.132 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-03-25T16:17:06.132 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-03-25T16:17:06.133 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:06.367 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 167 kB/s | 32 kB 00:00 2024-03-25T16:17:06.425 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 298 kB/s | 17 kB 00:00 2024-03-25T16:17:06.501 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-03-25T16:17:06.668 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-03-25T16:17:06.968 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T16:17:07.010 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T16:17:07.047 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:17:07.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:07.078 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:17:07.166 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:17:07.166 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:07.336 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.7 MB/s | 4.7 MB 00:00 2024-03-25T16:17:07.511 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-25T16:17:07.703 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T16:17:07.770 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-25T16:17:08.216 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:08.216 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:17:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:17:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:17:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:17:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:17:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:17:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:17:08.219 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:17:08.604 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 272 kB/s | 245 kB 00:00 2024-03-25T16:17:09.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:09.193 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:09.193 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-03-25T16:17:09.193 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.193 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.194 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.195 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:09.196 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:17:09.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-03-25T16:17:10.011 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:45 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:17:10.159 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:17:10.160 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-03-25T16:17:10.161 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:17:10.162 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 48 M 2024-03-25T16:17:10.162 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 204 M 2024-03-25T16:17:10.162 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:17:10.322 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-25T16:17:10.431 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.6 MB/s | 300 kB 00:00 2024-03-25T16:17:10.464 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:17:10.554 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.7 MB/s | 7.4 MB 00:04 2024-03-25T16:17:10.556 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:10.556 INFO:teuthology.orchestra.run.smithi171.stdout:Total 7.9 MB/s | 37 MB 00:04 2024-03-25T16:17:10.600 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:10.626 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:10.626 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:10.998 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:10.998 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:11.356 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 210 kB/s | 187 kB 00:00 2024-03-25T16:17:11.381 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 139 kB/s | 163 kB 00:01 2024-03-25T16:17:11.716 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:11.794 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:17:11.841 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:17:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:17:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:17:12.167 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:17:12.194 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:17:12.294 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:17:12.950 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:17:13.068 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:17:14.310 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:17:15.341 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:17:15.367 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:17:17.986 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:17:17.986 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:17:17.987 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:17:18.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:17:18.479 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:18.479 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:18.480 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:18.851 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:17:19.427 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:19.576 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:17:19.577 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:19.578 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 2.0 M 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 26 M 2024-03-25T16:17:19.579 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:19.747 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 252 kB/s | 24 kB 00:00 2024-03-25T16:17:19.797 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-25T16:17:19.847 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:17:19.873 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:17:19.898 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:17:19.923 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 867 kB/s | 43 kB 00:00 2024-03-25T16:17:20.215 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 92 kB/s | 52 kB 00:00 2024-03-25T16:17:20.399 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 64 kB/s | 32 kB 00:00 2024-03-25T16:17:20.425 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 30 kB/s | 15 kB 00:00 2024-03-25T16:17:20.608 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 324 kB/s | 59 kB 00:00 2024-03-25T16:17:20.792 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-03-25T16:17:20.934 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 282 kB/s | 203 kB 00:00 2024-03-25T16:17:21.801 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 746 kB/s | 1.0 MB 00:01 2024-03-25T16:17:21.805 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:21.805 INFO:teuthology.orchestra.run.smithi171.stdout:Total 942 kB/s | 2.0 MB 00:02 2024-03-25T16:17:21.806 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:21.820 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:21.820 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:21.915 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:21.916 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:22.607 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:17:22.662 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:17:22.719 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:17:22.756 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:17:22.797 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:17:22.892 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:17:22.993 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:17:23.049 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:17:23.775 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:17:23.831 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:17:23.865 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:17:23.914 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:17:23.952 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:17:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:17:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:17:25.174 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:17:25.174 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:25.174 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:25.174 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:25.174 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:25.175 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:25.421 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:17:25.989 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:26.139 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:26.140 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:26.140 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:17:26.140 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-03-25T16:17:26.141 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:26.142 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 6.8 M 2024-03-25T16:17:26.142 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 33 M 2024-03-25T16:17:26.142 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:26.510 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-03-25T16:17:26.594 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:17:26.894 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 219 kB/s | 138 kB 00:00 2024-03-25T16:17:26.961 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 157 kB/s | 58 kB 00:00 2024-03-25T16:17:27.136 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 877 kB/s | 548 kB 00:00 2024-03-25T16:17:27.236 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-25T16:17:27.240 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:27.240 INFO:teuthology.orchestra.run.smithi171.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-25T16:17:27.253 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:27.263 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:27.263 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:27.354 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:27.355 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:27.655 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:27.811 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:17:27.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:17:28.400 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:17:28.475 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:17:28.545 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:17:28.569 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:17:29.400 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:17:29.400 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:17:29.400 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:17:29.400 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:17:29.401 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:17:29.401 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:17:29.698 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:29.699 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:29.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2024-03-25T16:17:30.504 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:30.667 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:30.667 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:30.667 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-03-25T16:17:30.668 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:30.669 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 964 k 2024-03-25T16:17:30.669 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.8 M 2024-03-25T16:17:30.669 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:30.800 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 79 kB 00:00 2024-03-25T16:17:32.300 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 543 kB/s | 885 kB 00:01 2024-03-25T16:17:32.301 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:32.301 INFO:teuthology.orchestra.run.smithi171.stdout:Total 590 kB/s | 964 kB 00:01 2024-03-25T16:17:32.302 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:32.325 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:32.325 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:32.402 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:32.402 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:32.586 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:17:32.791 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:17:33.526 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:17:33.526 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:17:33.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:17:33.857 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:33.857 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:33.857 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:17:33.858 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:33.858 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:34.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-volume 2024-03-25T16:17:34.743 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:34.893 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:34.893 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:34.893 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-03-25T16:17:34.893 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-03-25T16:17:34.894 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:34.895 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 332 k 2024-03-25T16:17:34.895 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.5 M 2024-03-25T16:17:34.895 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:35.034 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 496 kB/s | 69 kB 00:00 2024-03-25T16:17:35.792 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 294 kB/s | 264 kB 00:00 2024-03-25T16:17:35.793 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:35.794 INFO:teuthology.orchestra.run.smithi171.stdout:Total 369 kB/s | 332 kB 00:00 2024-03-25T16:17:35.794 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:35.804 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:35.804 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:35.851 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:35.852 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:36.007 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:36.183 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:17:36.221 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:17:36.236 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:17:36.236 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:17:36.236 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:17:36.236 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:17:37.305 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:37.306 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:37.615 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2024-03-25T16:17:38.202 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:38.354 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 127 k 2024-03-25T16:17:38.355 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 456 k 2024-03-25T16:17:38.356 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:39.136 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 162 kB/s | 127 kB 00:00 2024-03-25T16:17:39.137 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:39.137 INFO:teuthology.orchestra.run.smithi171.stdout:Total 162 kB/s | 127 kB 00:00 2024-03-25T16:17:39.137 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:39.143 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:39.143 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:39.200 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:39.200 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:39.349 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:39.933 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:40.213 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:40.213 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:40.213 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:40.213 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:40.214 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:40.214 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:40.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2024-03-25T16:17:41.020 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:41.117 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:41.170 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:41.171 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:41.171 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:41.232 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2024-03-25T16:17:41.804 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:41.952 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:41.953 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:17:41.953 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:41.953 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 31 k 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 140 k 2024-03-25T16:17:41.954 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:42.326 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 84 kB/s | 31 kB 00:00 2024-03-25T16:17:42.327 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:42.327 INFO:teuthology.orchestra.run.smithi171.stdout:Total 83 kB/s | 31 kB 00:00 2024-03-25T16:17:42.327 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:42.331 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:42.331 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:42.365 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:42.366 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:42.496 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:42.609 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:43.005 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:43.273 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:43.428 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2024-03-25T16:17:43.999 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:44.096 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:44.148 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:44.149 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:44.149 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:44.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2024-03-25T16:17:44.767 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:44.864 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:44.915 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:44.916 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:44.916 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:44.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados 2024-03-25T16:17:45.531 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:09 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:45.629 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:45.682 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:45.683 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:45.683 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:45.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw 2024-03-25T16:17:46.336 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:10 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:46.439 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:46.493 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:46.494 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:46.494 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:46.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs 2024-03-25T16:17:47.122 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:47.217 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:47.271 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:47.272 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:47.272 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:47.313 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd 2024-03-25T16:17:47.877 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:47.973 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:17:48.025 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:48.026 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T16:17:48.026 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:48.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2024-03-25T16:17:48.638 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:48.785 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:48.786 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-03-25T16:17:48.786 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 85 k 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 231 k 2024-03-25T16:17:48.787 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:49.731 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 90 kB/s | 85 kB 00:00 2024-03-25T16:17:49.732 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:49.732 INFO:teuthology.orchestra.run.smithi171.stdout:Total 90 kB/s | 85 kB 00:00 2024-03-25T16:17:49.732 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:49.740 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:49.740 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:49.796 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:49.796 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:49.930 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:50.038 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:50.416 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:50.704 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:50.704 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:50.704 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:50.705 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:50.705 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:50.705 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:50.970 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2024-03-25T16:17:51.552 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:51.706 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.1 M 2024-03-25T16:17:51.707 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 13 M 2024-03-25T16:17:51.708 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:52.694 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 47 MB 00:42 2024-03-25T16:17:52.696 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:52.697 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.1 MB/s | 48 MB 00:42 2024-03-25T16:17:52.708 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:17:52.728 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:17:52.728 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:17:53.115 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:17:53.115 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:17:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:17:53.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:17:53.808 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:17:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:17:53.894 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:17:53.922 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:17:54.488 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-25T16:17:54.489 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:54.489 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-25T16:17:54.489 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:54.500 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:54.500 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:54.577 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:54.577 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:55.006 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:55.064 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:55.089 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:55.089 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:17:55.089 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:17:55.089 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:17:55.090 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:17:55.090 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:56.170 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:56.170 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:56.170 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:56.171 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:56.171 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:56.171 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:56.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2024-03-25T16:17:57.001 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:20 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:17:57.156 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:17:57.157 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:17:57.158 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:57.158 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 171 k 2024-03-25T16:17:57.158 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 494 k 2024-03-25T16:17:57.158 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:17:58.270 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 154 kB/s | 171 kB 00:01 2024-03-25T16:17:58.271 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:58.271 INFO:teuthology.orchestra.run.smithi171.stdout:Total 154 kB/s | 171 kB 00:01 2024-03-25T16:17:58.272 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:17:58.281 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:17:58.281 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:17:58.342 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:17:58.342 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:17:58.466 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:17:58.491 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:17:58.546 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:17:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:59.036 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:59.336 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:17:59.337 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:59.337 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:17:59.337 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:59.337 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:17:59.337 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:17:59.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:18:00.152 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:24 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:18:00.307 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:18:00.308 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:00.308 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:18:00.308 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:00.308 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:18:00.308 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 91 M 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 498 M 2024-03-25T16:18:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:18:00.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:18:01.242 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:01.243 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:01.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-03-25T16:18:02.164 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:02.318 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:02.320 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-03-25T16:18:02.320 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:18:02.320 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-03-25T16:18:02.320 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:02.320 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:18:02.321 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:18:02.322 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout:========================================================================================= 2024-03-25T16:18:02.323 INFO:teuthology.orchestra.run.smithi087.stdout:Install 36 Packages 2024-03-25T16:18:02.324 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:02.324 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 30 M 2024-03-25T16:18:02.324 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 107 M 2024-03-25T16:18:02.324 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:02.996 INFO:teuthology.orchestra.run.smithi087.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 12 kB/s | 6.6 kB 00:00 2024-03-25T16:18:06.246 INFO:teuthology.orchestra.run.smithi087.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 572 kB/s | 2.1 MB 00:03 2024-03-25T16:18:06.905 INFO:teuthology.orchestra.run.smithi087.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 342 kB/s | 1.5 MB 00:04 2024-03-25T16:18:07.127 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 3.3 MB/s | 22 MB 00:06 2024-03-25T16:18:07.230 INFO:teuthology.orchestra.run.smithi087.stdout:(4/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 4.7 MB 00:04 2024-03-25T16:18:07.355 INFO:teuthology.orchestra.run.smithi087.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:18:07.397 INFO:teuthology.orchestra.run.smithi087.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:18:07.431 INFO:teuthology.orchestra.run.smithi087.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-25T16:18:07.456 INFO:teuthology.orchestra.run.smithi087.stdout:(8/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 449 kB/s | 247 kB 00:00 2024-03-25T16:18:07.482 INFO:teuthology.orchestra.run.smithi087.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-25T16:18:07.507 INFO:teuthology.orchestra.run.smithi087.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-25T16:18:07.532 INFO:teuthology.orchestra.run.smithi087.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:18:07.582 INFO:teuthology.orchestra.run.smithi087.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:18:07.608 INFO:teuthology.orchestra.run.smithi087.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-25T16:18:07.633 INFO:teuthology.orchestra.run.smithi087.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:18:07.658 INFO:teuthology.orchestra.run.smithi087.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T16:18:07.733 INFO:teuthology.orchestra.run.smithi087.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2024-03-25T16:18:07.875 INFO:teuthology.orchestra.run.smithi087.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 516 kB/s | 125 kB 00:00 2024-03-25T16:18:07.967 INFO:teuthology.orchestra.run.smithi087.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 727 kB/s | 151 kB 00:00 2024-03-25T16:18:08.026 INFO:teuthology.orchestra.run.smithi087.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-03-25T16:18:08.051 INFO:teuthology.orchestra.run.smithi087.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:18:08.110 INFO:teuthology.orchestra.run.smithi087.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-25T16:18:08.168 INFO:teuthology.orchestra.run.smithi087.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T16:18:08.193 INFO:teuthology.orchestra.run.smithi087.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-25T16:18:08.227 INFO:teuthology.orchestra.run.smithi087.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-25T16:18:08.252 INFO:teuthology.orchestra.run.smithi087.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-25T16:18:08.286 INFO:teuthology.orchestra.run.smithi087.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-25T16:18:08.311 INFO:teuthology.orchestra.run.smithi087.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 790 kB/s | 46 kB 00:00 2024-03-25T16:18:08.345 INFO:teuthology.orchestra.run.smithi087.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:18:08.387 INFO:teuthology.orchestra.run.smithi087.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T16:18:08.412 INFO:teuthology.orchestra.run.smithi087.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-25T16:18:08.446 INFO:teuthology.orchestra.run.smithi087.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:18:08.471 INFO:teuthology.orchestra.run.smithi087.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-25T16:18:08.513 INFO:teuthology.orchestra.run.smithi087.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-25T16:18:08.555 INFO:teuthology.orchestra.run.smithi087.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-25T16:18:08.580 INFO:teuthology.orchestra.run.smithi087.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-25T16:18:23.221 INFO:teuthology.orchestra.run.smithi087.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-25T16:18:23.224 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:23.224 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.4 MB/s | 30 MB 00:20 2024-03-25T16:18:23.249 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:23.292 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:23.292 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:23.698 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:23.699 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:18:25.033 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:18:25.090 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:18:25.125 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:18:25.158 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:18:25.229 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:18:25.282 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:18:25.315 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:18:25.406 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:18:25.477 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:18:25.517 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:18:25.589 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:18:25.643 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:18:25.682 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:18:25.738 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:18:25.794 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:18:25.980 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:18:26.027 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:18:26.066 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:18:26.109 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:18:26.138 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:18:26.215 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:18:26.249 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:18:26.282 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:18:26.323 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:18:26.357 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:18:26.384 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:18:26.474 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:18:26.515 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:18:26.586 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:18:26.722 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:18:26.848 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:18:27.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:18:27.044 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:18:27.070 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:18:27.070 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:27.070 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:18:27.071 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:18:27.071 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:18:27.071 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:28.512 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:18:28.538 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:28.979 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:18:29.002 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:29.204 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:18:29.228 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:29.294 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:18:31.061 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:18:31.062 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:18:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:18:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:18:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:18:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:18:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:18:31.064 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:18:31.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:18:31.971 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:31.971 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:18:31.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:18:31.973 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:31.974 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:32.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-03-25T16:18:32.895 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:32.992 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:18:33.043 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:33.044 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:18:33.044 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:33.119 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-03-25T16:18:33.675 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:33.823 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:33.824 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:18:33.825 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:33.825 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 747 k 2024-03-25T16:18:33.825 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 754 k 2024-03-25T16:18:33.825 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:34.575 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 996 kB/s | 747 kB 00:00 2024-03-25T16:18:34.576 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:34.576 INFO:teuthology.orchestra.run.smithi087.stdout:Total 994 kB/s | 747 kB 00:00 2024-03-25T16:18:34.576 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:34.581 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:34.582 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:34.587 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:34.587 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:18:34.664 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:18:35.108 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:35.582 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:35.864 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:35.864 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:36.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:18:36.641 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:36.794 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 145 k 2024-03-25T16:18:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 436 k 2024-03-25T16:18:36.796 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:37.339 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 268 kB/s | 145 kB 00:00 2024-03-25T16:18:37.339 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:37.340 INFO:teuthology.orchestra.run.smithi087.stdout:Total 267 kB/s | 145 kB 00:00 2024-03-25T16:18:37.340 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:37.347 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:37.347 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:37.399 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:37.399 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:18:37.543 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:18:37.608 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:37.630 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:37.630 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:37.630 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:18:37.630 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:38.596 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:38.596 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:38.836 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-03-25T16:18:39.407 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:39.507 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:18:39.558 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:39.559 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:18:39.559 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:39.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:18:39.965 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 1.7 MB/s | 68 MB 00:39 2024-03-25T16:18:39.966 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:39.966 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.3 MB/s | 91 MB 00:39 2024-03-25T16:18:39.967 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:18:39.971 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:18:39.971 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:18:40.191 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:40.339 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:40.340 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-03-25T16:18:40.340 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:18:40.341 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout:====================================================================================== 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:40.342 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.9 M 2024-03-25T16:18:40.343 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-03-25T16:18:40.343 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:40.553 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:18:40.553 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:18:41.050 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 42 kB/s | 15 kB 00:00 2024-03-25T16:18:41.134 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 69 kB/s | 31 kB 00:00 2024-03-25T16:18:41.251 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T16:18:41.284 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-25T16:18:41.493 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-03-25T16:18:41.593 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T16:18:41.619 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-25T16:18:41.744 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-03-25T16:18:41.769 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 962 kB/s | 144 kB 00:00 2024-03-25T16:18:41.794 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-03-25T16:18:41.861 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T16:18:42.245 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.1 MB/s | 1.8 MB 00:01 2024-03-25T16:18:42.248 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:42.248 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-25T16:18:42.267 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:42.284 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:42.284 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:42.452 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:42.452 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:18:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:18:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:18:43.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:18:43.127 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:18:43.158 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:18:43.261 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:18:43.382 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:18:43.475 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:18:43.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:18:43.548 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:18:43.580 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:18:43.635 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:18:43.821 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:18:43.847 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:18:44.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:44.962 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:44.963 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:45.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:18:45.823 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:20 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:45.972 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:18:45.974 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-03-25T16:18:45.975 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:45.976 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2024-03-25T16:18:45.976 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 209 M 2024-03-25T16:18:45.976 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:46.409 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 32 kB 00:00 2024-03-25T16:18:46.442 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-25T16:18:46.484 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 820 kB/s | 34 kB 00:00 2024-03-25T16:18:46.659 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:18:46.793 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:18:46.852 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T16:18:46.877 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:18:47.244 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-25T16:18:47.344 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-25T16:18:47.386 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T16:18:47.528 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:18:50.211 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.9 MB/s | 7.4 MB 00:03 2024-03-25T16:18:50.214 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:50.214 INFO:teuthology.orchestra.run.smithi087.stdout:Total 8.7 MB/s | 37 MB 00:04 2024-03-25T16:18:50.258 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:50.283 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:50.283 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:50.641 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:18:50.811 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:18:51.165 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:18:51.365 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:18:51.434 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:18:51.465 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:18:51.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:18:51.602 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:18:51.772 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:18:51.801 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:18:51.880 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:18:52.552 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:18:52.672 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:18:53.905 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:18:54.566 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:18:54.567 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:54.826 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:18:54.932 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:18:54.958 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:18:55.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:18:55.621 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:18:55.781 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:18:55.782 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:18:55.783 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:18:55.783 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:18:55.783 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 416 M 2024-03-25T16:18:55.783 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.7 G 2024-03-25T16:18:55.783 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:18:57.341 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:18:57.341 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:18:57.341 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:18:57.341 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:18:57.341 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:18:57.773 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:57.774 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:18:58.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:18:58.726 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:18:58.884 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:18:58.885 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:18:58.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.0 M 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-03-25T16:18:58.887 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:18:59.149 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 173 kB/s | 18 kB 00:00 2024-03-25T16:18:59.191 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-25T16:18:59.241 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:18:59.266 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:18:59.291 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:18:59.317 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-25T16:18:59.434 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-25T16:18:59.459 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 194 kB/s | 32 kB 00:00 2024-03-25T16:18:59.601 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T16:18:59.659 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-25T16:18:59.685 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 82 kB/s | 52 kB 00:00 2024-03-25T16:18:59.735 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-25T16:18:59.760 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2024-03-25T16:18:59.762 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:59.762 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-25T16:18:59.762 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:18:59.775 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:18:59.775 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:18:59.864 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:18:59.865 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:00.378 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:00.552 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:19:00.603 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:19:00.658 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:19:00.704 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:19:00.745 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:19:00.840 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:19:00.940 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:19:00.988 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:19:01.706 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:19:01.760 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:19:01.794 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:19:01.843 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:19:01.897 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:19:02.641 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:19:02.641 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:19:02.641 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:19:02.642 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:19:03.002 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:19:03.002 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:19:03.003 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:19:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:19:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:03.004 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:03.276 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:19:03.844 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:38 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:03.994 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:19:03.995 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.8 M 2024-03-25T16:19:03.996 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-03-25T16:19:03.997 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:04.611 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:19:04.636 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 829 kB/s | 248 kB 00:00 2024-03-25T16:19:04.795 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 365 kB/s | 58 kB 00:00 2024-03-25T16:19:04.872 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 660 kB/s | 51 kB 00:00 2024-03-25T16:19:04.906 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:19:05.056 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 192 kB/s | 138 kB 00:00 2024-03-25T16:19:05.057 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:05.057 INFO:teuthology.orchestra.run.smithi087.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-25T16:19:05.071 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:05.081 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:05.081 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:05.171 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:05.171 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:05.457 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:05.653 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:19:05.698 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:19:06.210 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:19:06.293 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:19:06.354 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:19:06.377 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:19:07.198 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:19:07.198 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:19:07.198 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:19:07.199 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:19:07.199 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:19:07.199 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:19:07.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:07.485 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:07.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-03-25T16:19:08.303 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:08.454 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:08.458 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:08.459 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-03-25T16:19:08.459 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:08.459 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 964 k 2024-03-25T16:19:08.459 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.8 M 2024-03-25T16:19:08.459 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:08.682 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 348 kB/s | 79 kB 00:00 2024-03-25T16:19:09.423 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 915 kB/s | 885 kB 00:00 2024-03-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi087.stdout:Total 993 kB/s | 964 kB 00:00 2024-03-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:09.446 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:09.446 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:09.519 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:09.520 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:09.698 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:09.852 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:19:09.923 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:19:10.634 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:19:10.635 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:10.951 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:11.287 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-03-25T16:19:11.864 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:46 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:12.013 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:12.014 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-03-25T16:19:12.015 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:12.015 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 332 k 2024-03-25T16:19:12.015 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.5 M 2024-03-25T16:19:12.015 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:12.144 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 537 kB/s | 69 kB 00:00 2024-03-25T16:19:12.719 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 375 kB/s | 264 kB 00:00 2024-03-25T16:19:12.720 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:12.720 INFO:teuthology.orchestra.run.smithi087.stdout:Total 471 kB/s | 332 kB 00:00 2024-03-25T16:19:12.721 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:12.730 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:12.730 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:12.774 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:12.775 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:12.951 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:13.123 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:19:13.163 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:19:13.179 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:19:13.179 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:13.179 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:19:13.179 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:13.985 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:19:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:14.353 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:14.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-03-25T16:19:15.270 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:50 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:15.419 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:15.419 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:15.419 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 127 k 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2024-03-25T16:19:15.420 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:16.008 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 216 kB/s | 127 kB 00:00 2024-03-25T16:19:16.013 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:16.013 INFO:teuthology.orchestra.run.smithi087.stdout:Total 215 kB/s | 127 kB 00:00 2024-03-25T16:19:16.013 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:16.014 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:16.014 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:16.071 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:16.071 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:16.212 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:16.322 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:16.770 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:17.056 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:17.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-03-25T16:19:17.858 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:17.954 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:18.010 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:18.011 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:18.011 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:18.083 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-03-25T16:19:18.650 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:53 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:18.800 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:18.800 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:18.800 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 k 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 140 k 2024-03-25T16:19:18.801 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:19.310 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 61 kB/s | 31 kB 00:00 2024-03-25T16:19:19.311 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:19.311 INFO:teuthology.orchestra.run.smithi087.stdout:Total 61 kB/s | 31 kB 00:00 2024-03-25T16:19:19.311 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:19.315 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:19.315 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:19.348 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:19.348 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:19.484 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:19.584 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:19.959 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:20.214 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:20.215 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:20.215 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:20.215 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:20.215 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:20.215 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:20.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-03-25T16:19:20.962 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:55 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:21.060 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:21.112 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:21.113 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:21.113 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:21.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-03-25T16:19:21.733 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:56 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:21.830 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:21.882 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:21.883 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:21.883 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:21.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-03-25T16:19:22.515 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:57 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:22.610 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:22.662 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:22.662 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:22.663 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:22.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-03-25T16:19:23.298 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:58 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:23.396 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:23.448 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:23.449 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:23.449 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:23.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-03-25T16:19:24.077 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:24.173 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:24.224 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:24.225 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:24.225 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:24.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-03-25T16:19:24.838 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:24.935 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:24.986 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:24.987 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T16:19:24.987 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:25.043 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-03-25T16:19:25.610 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:00 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:25.760 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:25.761 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-03-25T16:19:25.762 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2024-03-25T16:19:25.762 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:26.363 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 142 kB/s | 85 kB 00:00 2024-03-25T16:19:26.364 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:26.364 INFO:teuthology.orchestra.run.smithi087.stdout:Total 141 kB/s | 85 kB 00:00 2024-03-25T16:19:26.364 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:26.371 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:26.371 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:26.429 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:26.429 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:26.567 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:26.670 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:27.060 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:27.339 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:27.339 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:27.339 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:27.340 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:27.340 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:27.340 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:27.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-03-25T16:19:28.182 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:03 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:28.334 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:19:28.335 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:28.336 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.1 M 2024-03-25T16:19:28.336 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 13 M 2024-03-25T16:19:28.336 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:31.228 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-25T16:19:31.229 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:31.230 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-25T16:19:31.230 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:31.240 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:31.240 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:31.316 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:31.317 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:31.757 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:31.817 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:31.842 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:31.842 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:31.842 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:19:31.842 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:19:31.842 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:19:31.843 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:32.982 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:32.982 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:32.982 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:32.982 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.982 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:32.983 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:33.226 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-03-25T16:19:33.813 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:08 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:33.962 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:33.964 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 171 k 2024-03-25T16:19:33.964 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 494 k 2024-03-25T16:19:33.964 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:19:34.886 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 186 kB/s | 171 kB 00:00 2024-03-25T16:19:34.886 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:34.887 INFO:teuthology.orchestra.run.smithi087.stdout:Total 186 kB/s | 171 kB 00:00 2024-03-25T16:19:34.887 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:19:34.895 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:19:34.896 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:19:34.954 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:19:34.955 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:19:35.103 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:19:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:35.661 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:35.948 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:19:36.177 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:19:36.750 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:11 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:19:36.905 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:19:36.905 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:36.905 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-03-25T16:19:36.906 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:19:36.907 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 91 M 2024-03-25T16:19:36.907 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 498 M 2024-03-25T16:19:36.907 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:20:13.843 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 622 kB/s | 22 MB 00:36 2024-03-25T16:21:31.310 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 610 kB/s | 68 MB 01:54 2024-03-25T16:21:31.310 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:31.311 INFO:teuthology.orchestra.run.smithi087.stdout:Total 811 kB/s | 91 MB 01:54 2024-03-25T16:21:31.311 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:21:31.316 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:21:31.316 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:21:31.900 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:21:31.900 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:21:35.006 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:21:42.178 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:21:42.319 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:21:45.549 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:21:45.549 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:21:45.804 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:45.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:21:45.804 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:21:45.804 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:45.804 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:45.805 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:21:45.805 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:21:46.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:21:46.579 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:21:46.728 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 416 M 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.7 G 2024-03-25T16:21:46.729 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:27:54.149 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 791 kB/s | 416 MB 08:58 2024-03-25T16:27:54.150 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:54.150 INFO:teuthology.orchestra.run.smithi171.stdout:Total 791 kB/s | 416 MB 08:58 2024-03-25T16:27:54.150 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:27:54.153 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:27:54.153 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:27:56.622 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:27:56.623 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:28:39.060 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:28:39.253 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:28:47.560 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:28:47.818 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:28:47.819 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:47.819 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:28:47.819 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:47.819 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:47.819 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:28:48.079 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:28:48.647 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:13:12 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:28:48.796 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:28:48.797 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:28:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:28:48.797 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:28:48.797 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================== 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.4 M 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 6.4 M 2024-03-25T16:28:48.798 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:28:55.931 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 205 kB/s | 1.4 MB 00:07 2024-03-25T16:28:55.932 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:55.932 INFO:teuthology.orchestra.run.smithi171.stdout:Total 205 kB/s | 1.4 MB 00:07 2024-03-25T16:28:55.932 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:28:55.937 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:28:55.937 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:28:55.980 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:28:55.980 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:28:56.233 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:28:56.329 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:28:56.752 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:57.018 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:28:57.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:28:57.739 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:13:21 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:28:57.887 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 176 M 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 753 M 2024-03-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:39:00.210 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 299 kB/s | 176 MB 10:02 2024-03-25T16:39:00.232 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:00.232 INFO:teuthology.orchestra.run.smithi171.stdout:Total 299 kB/s | 176 MB 10:02 2024-03-25T16:39:00.233 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T16:39:00.233 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T16:39:00.233 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T16:39:01.274 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T16:39:01.274 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T16:39:19.211 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T16:39:19.381 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:39:24.248 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:39:24.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:39:24.498 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:39:24.498 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T16:39:24.498 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:24.498 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:39:24.498 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T16:39:24.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:39:25.260 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:23:49 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T16:39:25.410 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:39:25.411 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 672 M 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 3.1 G 2024-03-25T16:39:25.412 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T16:45:05.573 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 305 kB/s | 416 MB 23:18 2024-03-25T16:45:05.574 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:05.574 INFO:teuthology.orchestra.run.smithi087.stdout:Total 305 kB/s | 416 MB 23:18 2024-03-25T16:45:05.574 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:45:05.577 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:45:05.577 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:45:08.041 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:45:08.041 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:45:50.112 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:45:50.209 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:58.132 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:58.387 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:58.387 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:45:58.388 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:45:58.388 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:58.388 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:45:58.388 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:45:58.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:45:59.191 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:29:34 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:45:59.338 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2024-03-25T16:45:59.339 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:45:59.340 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:45:59.340 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.4 M 2024-03-25T16:45:59.340 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 6.4 M 2024-03-25T16:45:59.340 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:46:06.254 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 211 kB/s | 1.4 MB 00:06 2024-03-25T16:46:06.255 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:46:06.255 INFO:teuthology.orchestra.run.smithi087.stdout:Total 211 kB/s | 1.4 MB 00:06 2024-03-25T16:46:06.255 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:46:06.260 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:46:06.260 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:46:06.305 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:46:06.306 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:46:06.558 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:46:06.661 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:46:07.084 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:46:07.358 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:46:07.563 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:46:08.127 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:29:43 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:46:08.275 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:46:08.275 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:46:08.275 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 176 M 2024-03-25T16:46:08.276 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 753 M 2024-03-25T16:46:08.277 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T16:57:57.853 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 253 kB/s | 176 MB 11:49 2024-03-25T16:57:57.854 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:57.854 INFO:teuthology.orchestra.run.smithi087.stdout:Total 253 kB/s | 176 MB 11:49 2024-03-25T16:57:57.854 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T16:57:57.857 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T16:57:57.857 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T16:57:58.916 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T16:57:58.916 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T16:58:17.190 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T16:58:17.365 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:21.980 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:58:22.245 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T16:58:22.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:58:23.011 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:41:58 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T16:58:23.158 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T16:58:23.159 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T16:58:23.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T16:58:23.160 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 672 M 2024-03-25T16:58:23.160 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 3.1 G 2024-03-25T16:58:23.160 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T17:19:51.646 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 284 kB/s | 672 MB 40:26 2024-03-25T17:19:51.690 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:51.690 INFO:teuthology.orchestra.run.smithi171.stdout:Total 284 kB/s | 672 MB 40:26 2024-03-25T17:19:51.690 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:19:51.690 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:19:51.690 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:19:55.609 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:19:55.651 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:21:06.485 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:21:06.678 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:21:16.750 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:16.992 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:21:17.275 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:21:17.844 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:05:41 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:21:17.941 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:21:17.993 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:21:17.994 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-03-25T17:21:17.994 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:21:18.055 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:21:18.621 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:05:42 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:21:18.771 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 20 M 2024-03-25T17:21:18.772 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 103 M 2024-03-25T17:21:18.773 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:21:54.959 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 557 kB/s | 20 MB 00:36 2024-03-25T17:21:54.959 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:54.959 INFO:teuthology.orchestra.run.smithi171.stdout:Total 557 kB/s | 20 MB 00:36 2024-03-25T17:21:54.960 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:21:54.963 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:21:54.963 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:21:55.112 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:21:55.113 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:21:57.343 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:21:57.458 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:58.533 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:58.786 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:58.787 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:58.787 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:21:58.787 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:58.787 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:58.787 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:21:58.946 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:21:59.524 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:06:23 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:21:59.684 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:21:59.684 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:59.684 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:21:59.684 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:59.684 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 48 M 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 173 M 2024-03-25T17:21:59.685 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:24:31.032 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 323 kB/s | 48 MB 02:31 2024-03-25T17:24:31.032 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:24:31.033 INFO:teuthology.orchestra.run.smithi171.stdout:Total 323 kB/s | 48 MB 02:31 2024-03-25T17:24:31.033 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:24:31.036 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:24:31.036 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:24:31.349 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:24:31.349 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:24:36.081 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:24:36.170 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:37.620 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:37.874 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:37.874 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:24:37.875 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:24:37.875 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:24:37.875 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:24:37.875 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:24:38.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:24:38.614 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:09:02 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:24:38.763 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:24:38.764 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:24:38.764 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:24:38.764 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:24:38.764 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:24:38.764 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 99 M 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 449 M 2024-03-25T17:24:38.765 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:29:51.214 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 323 kB/s | 99 MB 05:12 2024-03-25T17:29:51.214 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:29:51.215 INFO:teuthology.orchestra.run.smithi171.stdout:Total 323 kB/s | 99 MB 05:12 2024-03-25T17:29:51.215 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:29:51.218 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:29:51.218 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:29:51.827 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:29:51.827 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:30:02.104 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:30:02.210 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:05.211 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:05.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:05.463 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:30:05.463 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:30:05.463 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:05.463 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:30:05.464 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:30:05.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:30:06.225 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:14:30 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:30:06.374 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:30:06.374 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 287 M 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.2 G 2024-03-25T17:30:06.375 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:40:00.655 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 276 kB/s | 672 MB 41:37 2024-03-25T17:40:00.675 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:00.676 INFO:teuthology.orchestra.run.smithi087.stdout:Total 276 kB/s | 672 MB 41:37 2024-03-25T17:40:00.676 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T17:40:00.676 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T17:40:00.676 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T17:40:04.621 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T17:40:04.621 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T17:41:15.353 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T17:41:15.522 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:25.107 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:25.365 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:25.366 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:41:25.366 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T17:41:25.366 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:25.366 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:41:25.366 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T17:41:25.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:41:26.220 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:25:01 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T17:41:26.316 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:41:26.369 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T17:41:26.370 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-03-25T17:41:26.370 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T17:41:26.448 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:41:27.010 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:25:02 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T17:41:27.158 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T17:41:27.159 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:41:27.159 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T17:41:27.159 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 20 M 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 103 M 2024-03-25T17:41:27.160 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T17:42:32.128 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 310 kB/s | 20 MB 01:04 2024-03-25T17:42:32.129 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:42:32.129 INFO:teuthology.orchestra.run.smithi087.stdout:Total 310 kB/s | 20 MB 01:04 2024-03-25T17:42:32.129 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T17:42:32.133 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T17:42:32.133 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T17:42:32.282 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T17:42:32.282 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T17:42:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T17:42:34.601 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:35.529 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:35.788 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:42:35.788 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:42:35.788 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T17:42:35.788 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:35.789 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:42:35.789 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T17:42:35.976 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:42:36.540 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:26:11 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:42:36.688 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 48 M 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 173 M 2024-03-25T17:42:36.689 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T17:44:15.499 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 346 kB/s | 287 MB 14:09 2024-03-25T17:44:15.499 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:15.500 INFO:teuthology.orchestra.run.smithi171.stdout:Total 346 kB/s | 287 MB 14:09 2024-03-25T17:44:15.500 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:44:15.503 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:44:15.503 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:44:17.209 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:44:17.209 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:44:43.483 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 385 kB/s | 48 MB 02:06 2024-03-25T17:44:43.484 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:43.485 INFO:teuthology.orchestra.run.smithi087.stdout:Total 385 kB/s | 48 MB 02:06 2024-03-25T17:44:43.485 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T17:44:43.488 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T17:44:43.488 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T17:44:43.802 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T17:44:43.802 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T17:44:46.645 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:44:47.045 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:48.525 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T17:44:48.881 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:50.188 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:44:50.458 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T17:44:50.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:44:51.213 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:28:26 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T17:44:51.361 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T17:44:51.361 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 99 M 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 449 M 2024-03-25T17:44:51.362 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T17:44:55.020 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:55.289 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:55.290 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:44:55.290 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:44:55.290 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:55.290 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:44:55.290 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:44:55.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:44:56.060 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:29:20 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:44:56.209 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:44:56.210 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:44:56.210 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 5.9 M 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 28 M 2024-03-25T17:44:56.211 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:45:16.053 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 307 kB/s | 5.9 MB 00:19 2024-03-25T17:45:16.053 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:16.054 INFO:teuthology.orchestra.run.smithi171.stdout:Total 307 kB/s | 5.9 MB 00:19 2024-03-25T17:45:16.054 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:45:16.058 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:45:16.058 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:45:16.130 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:45:16.130 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:45:16.865 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:45:16.968 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:17.612 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:17.871 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:17.871 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:17.871 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:45:17.871 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:17.871 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:17.872 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:45:18.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:45:18.607 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:29:42 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:45:18.756 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:45:18.757 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 109 k 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 440 k 2024-03-25T17:45:18.758 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:45:19.622 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 126 kB/s | 109 kB 00:00 2024-03-25T17:45:19.622 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:19.623 INFO:teuthology.orchestra.run.smithi171.stdout:Total 126 kB/s | 109 kB 00:00 2024-03-25T17:45:19.623 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:45:19.627 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:45:19.627 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:45:19.664 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:45:19.664 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:45:19.785 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:45:19.887 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:45:20.254 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:20.496 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:45:20.655 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:45:21.222 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:29:45 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:45:21.374 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:45:21.375 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:21.376 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 5.4 M 2024-03-25T17:45:21.376 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 25 M 2024-03-25T17:45:21.376 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:45:29.615 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 665 kB/s | 5.4 MB 00:08 2024-03-25T17:45:29.616 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:29.617 INFO:teuthology.orchestra.run.smithi171.stdout:Total 665 kB/s | 5.4 MB 00:08 2024-03-25T17:45:29.617 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:45:29.622 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:45:29.622 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:45:29.689 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:45:29.689 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:45:30.351 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:45:30.446 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:45:31.002 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:45:31.293 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:45:31.293 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:31.293 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:45:31.294 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:31.294 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:31.294 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:45:31.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2-debuginfo 2024-03-25T17:45:32.045 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:29:56 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:45:32.195 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 44 M 2024-03-25T17:45:32.196 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 253 M 2024-03-25T17:45:32.197 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:47:03.198 INFO:teuthology.orchestra.run.smithi171.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 498 kB/s | 44 MB 01:30 2024-03-25T17:47:03.201 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:03.202 INFO:teuthology.orchestra.run.smithi171.stdout:Total 498 kB/s | 44 MB 01:31 2024-03-25T17:47:03.202 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:47:03.202 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:47:03.202 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:47:03.501 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:47:03.502 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:47:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:47:08.661 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:47:10.508 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:47:10.769 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:47:10.770 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:47:10.770 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:47:10.770 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:10.770 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:47:10.770 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:47:10.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:47:11.518 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:31:35 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:47:11.670 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:47:11.670 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:47:11.670 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:47:11.671 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:47:11.672 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 52 M 2024-03-25T17:47:11.672 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 308 M 2024-03-25T17:47:11.672 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:48:59.024 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 407 kB/s | 99 MB 04:07 2024-03-25T17:48:59.025 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:59.025 INFO:teuthology.orchestra.run.smithi087.stdout:Total 407 kB/s | 99 MB 04:07 2024-03-25T17:48:59.025 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T17:48:59.028 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T17:48:59.028 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T17:48:59.640 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T17:48:59.640 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T17:49:09.953 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T17:49:10.105 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:12.885 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:49:13.168 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T17:49:13.388 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:49:13.955 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:32:48 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:49:14.103 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 287 M 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.2 G 2024-03-25T17:49:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T17:49:31.738 INFO:teuthology.orchestra.run.smithi171.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 379 kB/s | 52 MB 02:20 2024-03-25T17:49:31.739 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:31.739 INFO:teuthology.orchestra.run.smithi171.stdout:Total 379 kB/s | 52 MB 02:20 2024-03-25T17:49:31.739 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:31.742 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:31.742 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:32.087 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:32.087 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:38.070 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:38.278 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:49:40.545 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:49:40.801 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:49:40.802 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:40.802 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:40.802 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:40.802 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:40.802 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:40.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:49:41.550 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:05 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:49:41.701 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:41.702 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:49:41.703 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:41.703 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 615 k 2024-03-25T17:49:41.703 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.1 M 2024-03-25T17:49:41.703 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:49:43.282 INFO:teuthology.orchestra.run.smithi171.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 389 kB/s | 615 kB 00:01 2024-03-25T17:49:43.282 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:43.283 INFO:teuthology.orchestra.run.smithi171.stdout:Total 389 kB/s | 615 kB 00:01 2024-03-25T17:49:43.283 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:43.287 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:43.287 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:43.328 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:43.329 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:43.500 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:43.607 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:44.018 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:44.267 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:49:44.268 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:44.268 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:44.268 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:44.268 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:44.268 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:44.440 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:49:45.007 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:09 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:49:45.158 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:49:45.158 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:45.158 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:49:45.159 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:45.160 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 771 k 2024-03-25T17:49:45.160 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.7 M 2024-03-25T17:49:45.160 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:49:46.701 INFO:teuthology.orchestra.run.smithi171.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 500 kB/s | 771 kB 00:01 2024-03-25T17:49:46.701 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:46.702 INFO:teuthology.orchestra.run.smithi171.stdout:Total 500 kB/s | 771 kB 00:01 2024-03-25T17:49:46.702 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:46.706 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:46.706 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:46.748 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:46.748 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:47.025 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:49:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:47.719 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:47.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:49:48.479 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:12 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:49:48.631 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:49:48.631 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.1 M 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 4.3 M 2024-03-25T17:49:48.632 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:49:51.562 INFO:teuthology.orchestra.run.smithi171.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 393 kB/s | 1.1 MB 00:02 2024-03-25T17:49:51.563 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:51.563 INFO:teuthology.orchestra.run.smithi171.stdout:Total 393 kB/s | 1.1 MB 00:02 2024-03-25T17:49:51.563 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:51.568 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:51.568 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:51.612 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:51.612 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:51.946 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:52.357 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:52.608 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:52.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:49:53.363 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:17 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:49:53.517 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:53.518 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 337 k 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.0 M 2024-03-25T17:49:53.519 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:49:54.473 INFO:teuthology.orchestra.run.smithi171.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 353 kB/s | 337 kB 00:00 2024-03-25T17:49:54.474 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:54.474 INFO:teuthology.orchestra.run.smithi171.stdout:Total 353 kB/s | 337 kB 00:00 2024-03-25T17:49:54.475 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:54.478 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:54.479 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:54.519 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:54.519 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:54.655 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:54.764 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:55.139 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:55.391 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:55.588 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:49:56.158 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:20 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:49:56.309 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:49:56.309 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:56.309 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:49:56.309 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:56.309 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:56.310 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 509 k 2024-03-25T17:49:56.311 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.5 M 2024-03-25T17:49:56.311 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:49:58.384 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 245 kB/s | 509 kB 00:02 2024-03-25T17:49:58.385 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:58.385 INFO:teuthology.orchestra.run.smithi171.stdout:Total 245 kB/s | 509 kB 00:02 2024-03-25T17:49:58.385 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:49:58.389 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:49:58.389 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:49:58.429 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:49:58.429 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:49:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:49:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:59.064 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:59.324 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:59.324 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:59.324 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:49:59.324 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:59.324 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:49:59.325 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:49:59.520 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:50:00.084 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:34:24 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:50:00.236 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:50:00.237 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:50:00.238 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:50:00.239 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 52 M 2024-03-25T17:50:00.239 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 298 M 2024-03-25T17:50:00.239 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:52:46.231 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 318 kB/s | 52 MB 02:45 2024-03-25T17:52:46.232 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:46.232 INFO:teuthology.orchestra.run.smithi171.stdout:Total 318 kB/s | 52 MB 02:45 2024-03-25T17:52:46.232 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:52:46.235 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:52:46.235 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:52:46.570 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:52:46.571 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:52:52.487 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:52:52.695 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:54.941 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:52:55.203 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:52:55.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:52:55.951 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 1:37:19 ago on Mon 25 Mar 2024 04:15:36 PM UTC. 2024-03-25T17:52:56.100 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-03-25T17:52:56.101 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-03-25T17:52:56.102 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:52:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.3 M 2024-03-25T17:52:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 5.5 M 2024-03-25T17:52:56.102 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-03-25T17:52:59.107 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 427 kB/s | 1.3 MB 00:03 2024-03-25T17:52:59.108 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:59.108 INFO:teuthology.orchestra.run.smithi171.stdout:Total 426 kB/s | 1.3 MB 00:03 2024-03-25T17:52:59.108 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-03-25T17:52:59.113 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-03-25T17:52:59.113 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-03-25T17:52:59.157 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-03-25T17:52:59.157 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-03-25T17:52:59.400 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-03-25T17:52:59.506 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:59.885 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:00.142 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:00.142 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:53:00.142 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-03-25T17:53:00.143 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:00.143 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T17:53:00.143 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-03-25T17:53:00.311 DEBUG:teuthology.parallel:result is None 2024-03-25T18:03:28.459 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 344 kB/s | 287 MB 14:14 2024-03-25T18:03:28.460 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:28.460 INFO:teuthology.orchestra.run.smithi087.stdout:Total 344 kB/s | 287 MB 14:14 2024-03-25T18:03:28.460 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:03:28.462 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:03:28.462 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:03:30.173 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:03:30.173 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:03:59.548 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:03:59.654 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:03.811 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:04.108 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:04.108 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:04.109 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:04:04.109 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:04.109 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:04.109 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:04:04.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:04:04.924 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:47:39 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:04:05.072 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:04:05.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:05.074 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.9 M 2024-03-25T18:04:05.074 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 28 M 2024-03-25T18:04:05.074 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:04:23.448 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 331 kB/s | 5.9 MB 00:18 2024-03-25T18:04:23.449 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:23.449 INFO:teuthology.orchestra.run.smithi087.stdout:Total 331 kB/s | 5.9 MB 00:18 2024-03-25T18:04:23.450 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:04:23.453 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:04:23.453 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:04:23.525 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:04:23.525 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:04:24.258 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:04:24.372 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:24.932 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:25.194 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:04:25.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:04:25.968 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:48:00 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:04:26.117 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 109 k 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 440 k 2024-03-25T18:04:26.119 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:04:26.818 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 155 kB/s | 109 kB 00:00 2024-03-25T18:04:26.819 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:26.819 INFO:teuthology.orchestra.run.smithi087.stdout:Total 155 kB/s | 109 kB 00:00 2024-03-25T18:04:26.819 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:04:26.822 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:04:26.822 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:04:26.857 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:04:26.858 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:04:26.977 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:04:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:27.479 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:27.744 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:27.744 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:27.744 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:04:27.744 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:27.745 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:27.745 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:04:27.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:04:28.523 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:48:03 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:04:28.671 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:28.672 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.4 M 2024-03-25T18:04:28.673 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 25 M 2024-03-25T18:04:28.673 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:04:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 417 kB/s | 5.4 MB 00:13 2024-03-25T18:04:41.812 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:41.813 INFO:teuthology.orchestra.run.smithi087.stdout:Total 417 kB/s | 5.4 MB 00:13 2024-03-25T18:04:41.813 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:04:41.817 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:04:41.817 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:04:41.885 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:04:41.885 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:04:42.545 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:04:42.653 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:43.170 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:43.433 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:04:43.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2-debuginfo 2024-03-25T18:04:44.212 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:48:19 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:04:44.359 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:04:44.360 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:04:44.361 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:04:44.361 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 44 M 2024-03-25T18:04:44.361 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 253 M 2024-03-25T18:04:44.361 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:07:02.200 INFO:teuthology.orchestra.run.smithi087.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 329 kB/s | 44 MB 02:17 2024-03-25T18:07:02.201 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:02.201 INFO:teuthology.orchestra.run.smithi087.stdout:Total 329 kB/s | 44 MB 02:17 2024-03-25T18:07:02.201 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:07:02.204 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:07:02.204 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:07:02.504 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:07:02.504 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:07:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:07:07.631 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:09.469 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:07:09.731 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:07:09.932 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:07:10.492 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:50:45 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:07:10.642 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:07:10.642 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:07:10.643 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:07:10.644 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 52 M 2024-03-25T18:07:10.644 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 308 M 2024-03-25T18:07:10.644 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:05.709 INFO:teuthology.orchestra.run.smithi087.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 303 kB/s | 52 MB 02:55 2024-03-25T18:10:05.709 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:05.709 INFO:teuthology.orchestra.run.smithi087.stdout:Total 303 kB/s | 52 MB 02:55 2024-03-25T18:10:05.709 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:05.712 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:05.712 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:06.054 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:06.054 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:12.056 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:12.239 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.316 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:14.605 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:14.804 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:10:15.358 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:53:50 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:15.508 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 615 k 2024-03-25T18:10:15.509 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.1 M 2024-03-25T18:10:15.510 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:19.286 INFO:teuthology.orchestra.run.smithi087.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 163 kB/s | 615 kB 00:03 2024-03-25T18:10:19.287 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:19.287 INFO:teuthology.orchestra.run.smithi087.stdout:Total 163 kB/s | 615 kB 00:03 2024-03-25T18:10:19.287 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:19.291 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:19.291 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:19.333 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:19.333 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:19.501 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:19.602 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:19.985 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:20.239 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:20.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:10:21.026 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:53:56 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:21.176 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 771 k 2024-03-25T18:10:21.177 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.7 M 2024-03-25T18:10:21.178 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:23.197 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 382 kB/s | 771 kB 00:02 2024-03-25T18:10:23.198 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:23.198 INFO:teuthology.orchestra.run.smithi087.stdout:Total 381 kB/s | 771 kB 00:02 2024-03-25T18:10:23.198 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:23.202 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:23.202 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:23.243 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:23.244 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:23.427 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:23.537 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:23.914 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:24.172 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:24.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:24.172 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:24.172 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:24.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:24.173 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:24.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:10:24.933 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:53:59 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:25.082 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:25.082 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:25.082 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:25.083 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:25.084 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.1 M 2024-03-25T18:10:25.084 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 4.3 M 2024-03-25T18:10:25.084 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:29.100 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 287 kB/s | 1.1 MB 00:04 2024-03-25T18:10:29.101 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:29.101 INFO:teuthology.orchestra.run.smithi087.stdout:Total 286 kB/s | 1.1 MB 00:04 2024-03-25T18:10:29.102 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:29.106 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:29.106 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:29.148 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:29.149 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:29.369 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:29.474 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:29.850 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:30.111 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:30.111 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:30.112 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:30.112 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:30.112 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:30.112 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:30.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:10:30.896 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:54:05 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:31.046 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:31.047 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 337 k 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.0 M 2024-03-25T18:10:31.048 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:32.895 INFO:teuthology.orchestra.run.smithi087.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 183 kB/s | 337 kB 00:01 2024-03-25T18:10:32.895 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:32.896 INFO:teuthology.orchestra.run.smithi087.stdout:Total 182 kB/s | 337 kB 00:01 2024-03-25T18:10:32.896 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:32.900 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:32.900 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:32.939 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:32.939 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:33.081 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:33.194 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:33.804 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:34.061 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:10:34.623 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:54:09 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:34.771 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:34.772 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 509 k 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.5 M 2024-03-25T18:10:34.773 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:10:36.806 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 250 kB/s | 509 kB 00:02 2024-03-25T18:10:36.806 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:36.807 INFO:teuthology.orchestra.run.smithi087.stdout:Total 250 kB/s | 509 kB 00:02 2024-03-25T18:10:36.807 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:10:36.811 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:10:36.811 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:10:36.850 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:10:36.851 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:10:37.009 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:10:37.122 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:37.761 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:37.762 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:37.762 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:10:37.762 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:37.762 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:37.762 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:10:37.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:10:38.527 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:54:13 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:38.676 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 52 M 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 298 M 2024-03-25T18:10:38.677 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:13:18.943 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 329 kB/s | 52 MB 02:40 2024-03-25T18:13:18.943 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:18.944 INFO:teuthology.orchestra.run.smithi087.stdout:Total 329 kB/s | 52 MB 02:40 2024-03-25T18:13:18.944 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:13:18.946 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:13:18.947 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:13:19.280 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:13:19.281 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:13:25.186 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:13:25.290 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:27.299 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:27.572 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:27.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:27.572 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:13:27.572 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:27.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:27.573 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:13:27.773 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:13:28.339 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 1:57:03 ago on Mon 25 Mar 2024 04:16:25 PM UTC. 2024-03-25T18:13:28.487 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:13:28.488 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.3 M 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 5.5 M 2024-03-25T18:13:28.489 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-25T18:13:34.744 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 205 kB/s | 1.3 MB 00:06 2024-03-25T18:13:34.745 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:34.745 INFO:teuthology.orchestra.run.smithi087.stdout:Total 205 kB/s | 1.3 MB 00:06 2024-03-25T18:13:34.745 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-25T18:13:34.749 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-25T18:13:34.749 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-25T18:13:34.793 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-25T18:13:34.794 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-25T18:13:35.056 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-25T18:13:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:35.618 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:35.888 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-25T18:13:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:35.889 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-25T18:13:36.138 DEBUG:teuthology.parallel:result is None 2024-03-25T18:13:36.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:36.361 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:36.381 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:36.382 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:36.382 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:36.384 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:36.600 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:36.625 INFO:teuthology.orchestra.run.smithi171.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:36.625 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:36.625 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:36.627 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:13:36.627 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:36.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:36.655 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:36.655 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:36.697 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:13:36.697 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:36.697 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:36.724 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:36.791 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:36.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:36.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:36.886 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:13:36.887 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:36.887 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:36.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:36.979 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:36.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:37.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:37.074 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:13:37.074 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:37.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:37.100 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:37.167 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:37.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:37.194 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:37.261 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:13:37.344 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:13:37.345 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:37.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:37.374 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:13:37.374 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:37.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:37.442 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:13:37.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:37.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:37.529 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:13:37.529 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:13:37.529 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621247', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIgV9nVxjA/tkjSmI2/rMBJUsU3ZgOjTL1q0uxeQIB2jJz7E19yBi0JwhUL3MOe4vvRZsUDnne1+evrxO55qos0=', 'smithi171.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBt6g2fh+YzZnvAAW6vAoxLl/uWoK/Ld3GV5Mvn9yC9qpXLYU65zgeGJ1QY6FY0NWTToyw6y3UDkkKKlSlz75Ec='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:13:37.530 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:37.548 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:37.564 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:37.603 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:37.631 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:37.631 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:37.681 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:37.681 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-25 16:19:32.089774806 +0000 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-25 16:14:11.535419518 +0000 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-25 16:14:11.535419518 +0000 2024-03-25T18:13:37.736 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-25 16:14:11.535419518 +0000 2024-03-25T18:13:37.736 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:37.801 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-25T18:13:37.802 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-25T18:13:37.802 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000195477 s, 2.6 MB/s 2024-03-25T18:13:37.803 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:37.859 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-25 16:19:32.090774783 +0000 2024-03-25T18:13:37.914 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-25 16:14:11.875411614 +0000 2024-03-25T18:13:37.915 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-25 16:14:11.875411614 +0000 2024-03-25T18:13:37.915 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-25 16:14:11.875411614 +0000 2024-03-25T18:13:37.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:37.979 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-25T18:13:37.980 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-25T18:13:37.980 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000191835 s, 2.7 MB/s 2024-03-25T18:13:37.981 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:38.037 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-25 16:19:32.090774783 +0000 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-25 16:14:12.223403523 +0000 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-25 16:14:12.223403523 +0000 2024-03-25T18:13:38.091 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-25 16:14:12.223403523 +0000 2024-03-25T18:13:38.091 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:38.156 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-25T18:13:38.156 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-25T18:13:38.156 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000186826 s, 2.7 MB/s 2024-03-25T18:13:38.158 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:38.213 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:38.266 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-25 16:19:32.090774783 +0000 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-25 16:14:12.577395293 +0000 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-25 16:14:12.577395293 +0000 2024-03-25T18:13:38.267 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-25 16:14:12.577395293 +0000 2024-03-25T18:13:38.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:38.331 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-25T18:13:38.331 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-25T18:13:38.331 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000186032 s, 2.8 MB/s 2024-03-25T18:13:38.332 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:38.388 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:38.388 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:38.389 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:38.406 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:38.406 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-25 16:17:55.321760417 +0000 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-25 16:14:36.402316922 +0000 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-25 16:14:36.402316922 +0000 2024-03-25T18:13:38.461 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-25 16:14:36.402316922 +0000 2024-03-25T18:13:38.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:38.527 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-25T18:13:38.527 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-25T18:13:38.527 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000244453 s, 2.1 MB/s 2024-03-25T18:13:38.528 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:38.584 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:38.639 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:38.639 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.639 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:13:38.639 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.640 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.640 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-25 16:17:55.321760417 +0000 2024-03-25T18:13:38.640 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-25 16:14:36.748310030 +0000 2024-03-25T18:13:38.640 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-25 16:14:36.748310030 +0000 2024-03-25T18:13:38.640 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-25 16:14:36.748310030 +0000 2024-03-25T18:13:38.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:38.705 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-25T18:13:38.705 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-25T18:13:38.705 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000204434 s, 2.5 MB/s 2024-03-25T18:13:38.706 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:38.762 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-25 16:17:55.321760417 +0000 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-25 16:14:37.076303497 +0000 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-25 16:14:37.076303497 +0000 2024-03-25T18:13:38.817 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-25 16:14:37.076303497 +0000 2024-03-25T18:13:38.817 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:38.882 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-25T18:13:38.882 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-25T18:13:38.883 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000206946 s, 2.5 MB/s 2024-03-25T18:13:38.884 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:38.940 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-03-25 16:17:55.322760398 +0000 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-03-25 16:14:37.417296705 +0000 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-03-25 16:14:37.417296705 +0000 2024-03-25T18:13:38.995 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-03-25 16:14:37.417296705 +0000 2024-03-25T18:13:38.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:39.060 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-03-25T18:13:39.060 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-03-25T18:13:39.061 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000193289 s, 2.6 MB/s 2024-03-25T18:13:39.062 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:39.117 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:39.118 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:39.118 INFO:tasks.ceph:Generating config... 2024-03-25T18:13:39.119 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:13:39.119 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:13:39.119 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:13:39.119 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:13:39.119 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:13:39.119 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:13:39.120 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:13:39.120 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:13:39.120 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:13:39.120 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:13:39.120 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:13:39.121 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:13:39.121 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:13:39.121 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:13:39.122 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:13:39.123 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:13:39.123 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:13:39.123 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:13:39.123 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:13:39.123 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:13:39.123 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:39.182 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:13:39.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:13:39.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:39.256 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.87:6789]'), ('mon.c', '[v1:172.21.15.87:6790]'), ('mon.b', '[v1:172.21.15.171:6789]')] 2024-03-25T18:13:39.257 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.171:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:13:39.257 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:39.257 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:39.311 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.87:6789]' --addv c '[v1:172.21.15.87:6790]' --addv b '[v1:172.21.15.171:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 17 (quincy) 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 3 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:13:39.397 INFO:teuthology.orchestra.run.smithi087.stdout:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:13:39.398 INFO:teuthology.orchestra.run.smithi087.stdout:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:13:39.398 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:13:39.399 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:39.413 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 97b2a0dc-69f2-4b94-95e9-3722e689d10f... 2024-03-25T18:13:39.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:39.456 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:39.504 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.505 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.506 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = "[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.171:6789]" 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: mon election default strategy = 3 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr2 = False 2024-03-25T18:13:39.507 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: ms type = async 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd heartbeat grace = 80 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:39.508 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = stupid 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore zero block detection = True 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1 2024-03-25T18:13:39.510 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd class default list = * 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd class load list = * 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd client watch timeout = 120 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:39.511 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify missing on start = True 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd fast shutdown = False 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd max markdown count = 1000 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue = debug_random 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-03-25T18:13:39.512 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:39.513 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:39.514 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd crush smoke test = False 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon scrub interval = 300 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on pool no app = False 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-03-25T18:13:39.515 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout: debug objecter = 20 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout: debug rados = 20 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.c] 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.b] 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2024-03-25T18:13:39.516 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2024-03-25T18:13:39.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:39.518 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2024-03-25T18:13:39.519 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "[v1:172.21.15.87:6789],[v1:172.21.15.87:6790],[v1:172.21.15.171:6789]" 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: mon client directed command retry = 5 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: mon election default strategy = 3 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: ms bind msgr2 = False 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: ms inject socket failures = 5000 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: ms type = async 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: osd heartbeat grace = 80 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2024-03-25T18:13:39.520 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug trim objects = true 2024-03-25T18:13:39.521 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:39.522 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bdev async discard = True 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bdev enable discard = True 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore allocator = stupid 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore zero block detection = True 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 1 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 1 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: debug monc = 20 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-25T18:13:39.523 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd class default list = * 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd class load list = * 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd client watch timeout = 120 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify missing on start = True 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd fast shutdown = False 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd max markdown count = 1000 2024-03-25T18:13:39.524 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue = debug_random 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:39.525 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:39.526 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: # 1m isn't quite enough 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd crush smoke test = False 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:39.527 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: mon scrub interval = 300 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on pool no app = False 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: paxos service trim min = 10 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: debug monc = 20 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: debug objecter = 20 2024-03-25T18:13:39.528 INFO:teuthology.orchestra.run.smithi171.stdout: debug rados = 20 2024-03-25T18:13:39.529 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2024-03-25T18:13:39.529 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2024-03-25T18:13:39.529 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2024-03-25T18:13:39.530 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:13:39.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:13:39.585 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:13:39.585 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:39.585 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:13:39.600 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:39.600 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:13:39.654 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:13:39.655 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:39.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:39.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:39.733 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:13:39.733 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.788 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:13:39.788 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:39.788 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:39.788 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:39.828 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:13:39.828 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.883 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:13:39.883 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:13:39.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:39.938 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:39.940 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:40.006 INFO:teuthology.orchestra.run.smithi171.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:40.008 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:13:40.008 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:13:40.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:40.051 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:40.066 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:40.109 INFO:teuthology.orchestra.run.smithi171.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:40.124 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:13:40.124 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:40.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:40.150 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:40.151 INFO:tasks.ceph:role: osd.0 2024-03-25T18:13:40.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:13:40.151 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:40.224 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:40.745 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-03-25T18:13:40.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-03-25T18:13:40.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:40.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:40.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:40.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:40.967 INFO:tasks.ceph:role: osd.1 2024-03-25T18:13:40.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:13:40.968 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:41.044 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:41.044 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:41.044 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:41.045 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:41.046 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:41.500 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-03-25T18:13:41.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-03-25T18:13:41.503 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:41.596 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:41.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:41.697 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:41.697 INFO:tasks.ceph:role: osd.2 2024-03-25T18:13:41.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:13:41.698 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:41.770 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:41.771 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:41.772 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:41.772 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:42.269 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-03-25T18:13:42.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-03-25T18:13:42.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:42.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:42.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:42.529 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:42.529 INFO:tasks.ceph:role: osd.3 2024-03-25T18:13:42.529 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:13:42.529 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:42.603 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:42.603 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:42.603 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:42.603 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:42.604 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:43.127 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-03-25T18:13:43.129 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-03-25T18:13:43.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:43.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:43.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.407+0000 7f31c6523640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.407+0000 7f31c6523640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.407+0000 7f31c6523640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.407+0000 7f31c6523640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.407+0000 7f31c6523640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:43.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:43.408+0000 7f31c6523640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:13:46.245 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:46.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.361+0000 7fc311c89640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.361+0000 7fc311c89640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.361+0000 7fc311c89640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.361+0000 7fc311c89640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.361+0000 7fc311c89640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:46.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:46.362+0000 7fc311c89640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:13:49.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:49.235 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:49.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.327+0000 7f240e391640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:13:49.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.327+0000 7f240e391640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:13:49.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.327+0000 7f240e391640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:49.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.327+0000 7f240e391640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:49.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.327+0000 7f240e391640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:49.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:49.328+0000 7f240e391640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:13:52.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:52.193 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.281+0000 7f80a8914640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.281+0000 7f80a8914640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.281+0000 7f80a8914640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.281+0000 7f80a8914640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.281+0000 7f80a8914640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:13:52.282+0000 7f80a8914640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:13:55.122 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:55.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:55.184 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:55.184 INFO:tasks.ceph:role: osd.4 2024-03-25T18:13:55.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:13:55.184 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:55.262 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:55.262 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:55.262 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:55.262 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:55.263 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:55.761 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-25T18:13:55.763 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-25T18:13:55.763 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:55.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:55.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:55.982 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:55.982 INFO:tasks.ceph:role: osd.5 2024-03-25T18:13:55.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:13:55.982 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:56.057 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:56.057 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:56.057 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:56.057 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:56.058 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:56.524 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-25T18:13:56.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-25T18:13:56.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:56.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:56.646 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:56.714 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:56.714 INFO:tasks.ceph:role: osd.6 2024-03-25T18:13:56.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:13:56.714 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:57.273 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-25T18:13:57.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-25T18:13:57.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:57.423 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:57.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:57.528 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:57.528 INFO:tasks.ceph:role: osd.7 2024-03-25T18:13:57.528 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:13:57.528 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:57.604 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:57.605 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:58.124 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-03-25T18:13:58.126 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-03-25T18:13:58.126 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:58.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:58.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:58.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.422+0000 7f767880d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:13:58.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.423+0000 7f767880d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:13:58.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.423+0000 7f767880d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:58.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.423+0000 7f767880d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:58.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.423+0000 7f767880d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:58.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:13:58.423+0000 7f767880d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:14:01.267 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:14:01.299 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:14:01.390 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.389+0000 7f5895ac8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:14:01.390 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.389+0000 7f5895ac8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:14:01.390 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.389+0000 7f5895ac8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:14:01.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.389+0000 7f5895ac8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:14:01.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.389+0000 7f5895ac8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:14:01.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:01.390+0000 7f5895ac8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:14:04.231 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:14:04.263 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:14:04.354 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.353+0000 7f04eb0b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:14:04.354 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.354+0000 7f04eb0b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:14:04.354 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.354+0000 7f04eb0b3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:14:04.354 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.354+0000 7f04eb0b3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:14:04.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.354+0000 7f04eb0b3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:14:04.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:04.354+0000 7f04eb0b3640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:14:07.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:14:07.226 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.317+0000 7fc0b1a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.318+0000 7fc0b1a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.318+0000 7fc0b1a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.318+0000 7fc0b1a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.318+0000 7fc0b1a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:14:07.319 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:07.318+0000 7fc0b1a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:14:10.161 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:14:10.192 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:14:10.192 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:14:10.225 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:14:10.295 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:14:10.365 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:14:10.434 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:14:10.502 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:14:10.534 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:14:10.604 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:14:10.673 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.674 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:14:10.742 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.742 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:14:10.810 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:10.810 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:14:10.827 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:10.827 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:14:10.866 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:14:10.866 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:14:10.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.y] 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDTvgFm+VjtNxAAe67Upab3j+kHSIhQ8bvNNA== 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDXvgFmjZFhGBAA7HYY+VwXg9pScP9Ux+MDXA== 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-03-25T18:14:10.909 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDavgFm2gKlFRAArd60SwSPNsxld7SajGnrYw== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDdvgFmmDmTExAABlvBQLfCcEyunbq23mdZdg== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.3] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDgvgFmy5nPEBAAlEH+K5B4IInWNAYsrZuQ5w== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDUvgFmPLFlABAATduaSRkkdKipNxKmAn9hQA== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.4] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDmvgFmu8VPGRAArgKccKFKgMTk7hGzKqk0OQ== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.5] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDpvgFmIMNHFxAAwi+sY7WblfMB81v5hSCVcg== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.6] 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDsvgFmoOAjFRAAcEZSBZ5JSS1IIiWx/uhX+w== 2024-03-25T18:14:10.910 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.7] 2024-03-25T18:14:10.911 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDvvgFmipkBExAArYoFBG9Tme9mTz2YaHGq3Q== 2024-03-25T18:14:10.911 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-03-25T18:14:10.911 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDUvgFmuVcQAxAAo0y7Q0nEjfjYoZr3WSW8tw== 2024-03-25T18:14:10.911 INFO:teuthology.orchestra.run.smithi087.stdout:[client.1] 2024-03-25T18:14:10.911 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQDUvgFmJeCFBhAAUPaSnT1t+1f+yefcI51Naw== 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.y] 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDTvgFm+VjtNxAAe67Upab3j+kHSIhQ8bvNNA== 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.0] 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDXvgFmjZFhGBAA7HYY+VwXg9pScP9Ux+MDXA== 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.1] 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDavgFm2gKlFRAArd60SwSPNsxld7SajGnrYw== 2024-03-25T18:14:10.932 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.2] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDdvgFmmDmTExAABlvBQLfCcEyunbq23mdZdg== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.3] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDgvgFmy5nPEBAAlEH+K5B4IInWNAYsrZuQ5w== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.x] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDUvgFmPLFlABAATduaSRkkdKipNxKmAn9hQA== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.4] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDmvgFmu8VPGRAArgKccKFKgMTk7hGzKqk0OQ== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.5] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDpvgFmIMNHFxAAwi+sY7WblfMB81v5hSCVcg== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.6] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDsvgFmoOAjFRAAcEZSBZ5JSS1IIiWx/uhX+w== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.7] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDvvgFmipkBExAArYoFBG9Tme9mTz2YaHGq3Q== 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout:[client.0] 2024-03-25T18:14:10.933 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDUvgFmuVcQAxAAo0y7Q0nEjfjYoZr3WSW8tw== 2024-03-25T18:14:10.934 INFO:teuthology.orchestra.run.smithi171.stdout:[client.1] 2024-03-25T18:14:10.934 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQDUvgFmJeCFBhAAUPaSnT1t+1f+yefcI51Naw== 2024-03-25T18:14:10.934 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:14:10.954 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:14:11.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.081 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.083 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.134 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.191 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:14:11.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:14:11.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.403 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.405 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.459 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:14:11.511 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:14:11.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:14:11.567 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:14:11.569 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:14:11.624 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:14:11.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:14:11.651 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:14:11.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:14:11.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:14:11.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:14:11.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:14:11.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:14:11.992 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:14:12.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:14:12.118 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:14:12.120 DEBUG:teuthology.orchestra.run.smithi171:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:14:12.173 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:14:12.173 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:14:12.173 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:14:12.174 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:14:12.176 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:14:12.176 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:14:12.176 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:14:12.176 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:14:12.178 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:14:12.178 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:14:12.178 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:14:12.178 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:14:12.215 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:14:12.216 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:14:12.216 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:14:12.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:14:12.218 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:14:12.218 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:14:12.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:14:12.220 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:14:12.220 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:14:12.220 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.331+0000 7fdacd04a640 10 monclient: get_monmap_and_config 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.331+0000 7fdacd04a640 10 monclient: build_initial_monmap 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.331+0000 7fdacd04a640 10 monclient: monmap: 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:14:12.332800+0000 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:14:12.332800+0000 2024-03-25T18:14:12.332 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:12.333 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:12.333 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:12.333 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:12.333 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:12.333 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:12.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.332+0000 7fdacd04a640 1 Processor -- start 2024-03-25T18:14:12.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.332+0000 7fdacd04a640 1 -- start start 2024-03-25T18:14:12.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.332+0000 7fdacd04a640 10 monclient: init 2024-03-25T18:14:12.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-c con 0x7fdac80a9920 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-a con 0x7fdac8146450 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac806b100 con 0x7fdac8146450 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80b2940 con 0x7fdac80a9920 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac813fec0 con 0x7fdac80b45a0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdacd04a640 10 monclient(hunting): authenticate will time out at 7767.628057s 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdac8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdac814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:12.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.333+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdac813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:12.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.534+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdac814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:12.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.534+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdac8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:12.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.534+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdac813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:12.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.935+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdac8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:12.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.935+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdac814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:12.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:12.935+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdac813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:13.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:13.736+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdac8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:13.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:13.736+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdac814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:13.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:13.736+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdac813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:15.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:15.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:15.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:15.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdac8145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdac813f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdac814dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdac80a9920 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdac8146450 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac806b100 con 0x7fdac8146450 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80b2940 con 0x7fdac80a9920 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac813fec0 con 0x7fdac80b45a0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:15.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.333+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab40056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:15.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.534+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:15.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.534+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:15.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.534+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab40056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:15.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.935+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:15.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.935+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:15.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:15.935+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab40056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:16.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:16.736+0000 7fdac659c640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:16.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:16.736+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:16.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:16.736+0000 7fdac6d9d640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab40056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab40056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdac80a9920 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdac8146450 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4010e20 con 0x7fdac8146450 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac806b100 con 0x7fdac80a9920 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80b2940 con 0x7fdac80b45a0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac6d9d640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.333+0000 7fdac659c640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:18.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.534+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:18.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.534+0000 7fdac6d9d640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:18.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.534+0000 7fdac659c640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:18.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.935+0000 7fdac6d9d640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:18.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.935+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:18.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:18.935+0000 7fdac659c640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:19.191 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:00:06.870 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.299 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:00:07.012 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.301 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:00:07.014 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.498 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:00:07.177 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.619 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:00:07.332 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.621 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:00:07.334 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:19.736+0000 7fdac759e640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6789/0 2024-03-25T18:14:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:19.736+0000 7fdac6d9d640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.87:6790/0 2024-03-25T18:14:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:19.736+0000 7fdac659c640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-03-25T18:14:21.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:21.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4005630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdac80a9920 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdac8146450 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac813fec0 con 0x7fdac8146450 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4010e20 con 0x7fdac80a9920 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac806b100 con 0x7fdac80b45a0 2024-03-25T18:14:21.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.333+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:21.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.378+0000 7fdac6d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 0x7fdab4009670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:35166/0 (socket says 172.21.15.87:35166) 2024-03-25T18:14:21.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:21.378+0000 7fdac6d9d640 1 -- 172.21.15.87:0/3290102026 learned_addr learned my addr 172.21.15.87:0/3290102026 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdac80a9920 legacy=0x7fdab4009670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdac8146450 legacy=0x7fdab4008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab4005700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab4009670 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:24.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80b2940 con 0x7fdab4009670 2024-03-25T18:14:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4009a50 con 0x7fdab4005700 2024-03-25T18:14:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40075a0 con 0x7fdac80b45a0 2024-03-25T18:14:24.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:24.333+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:27.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab400c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4005700 legacy=0x7fdab4009dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab4009670 legacy=0x7fdab4005e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab40088f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab4005700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdac80b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4008740 con 0x7fdab40088f0 2024-03-25T18:14:27.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab400b6c0 con 0x7fdab4005700 2024-03-25T18:14:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab400c900 con 0x7fdac80b45a0 2024-03-25T18:14:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:27.333+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdac80b45a0 legacy=0x7fdab4005e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4005700 legacy=0x7fdab4009dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab40088f0 legacy=0x7fdab4008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdab400b850 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab400c3a0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab40088f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab400db40 con 0x7fdab40088f0 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4009ec0 con 0x7fdab400c3a0 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab400a0f0 con 0x7fdab400b850 2024-03-25T18:14:30.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:30.334+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdab400b850 legacy=0x7fdab400bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab400c3a0 legacy=0x7fdab4005e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab40088f0 legacy=0x7fdab4008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdab400caf0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab400d6a0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab4005e90 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.334+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab400a320 con 0x7fdab4005e90 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4007f30 con 0x7fdab400d6a0 2024-03-25T18:14:33.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40080c0 con 0x7fdab400caf0 2024-03-25T18:14:33.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:33.335+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:36.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:36.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:36.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:36.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:36.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdab400caf0 legacy=0x7fdab400cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab400d6a0 legacy=0x7fdab400bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab4005e90 legacy=0x7fdab4007820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdab4008d30 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab4006ca0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab4005e90 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4008250 con 0x7fdab4005e90 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40085a0 con 0x7fdab4006ca0 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4009160 con 0x7fdab4008d30 2024-03-25T18:14:36.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.335+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:36.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.336+0000 7fdac659c640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 legacy=0x7fdab40190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:14:36.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.336+0000 7fdac659c640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 legacy=0x7fdab40190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:14:36.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:36.336+0000 7fdac659c640 1 --1- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 0x7fdab40190c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T18:14:39.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): tick 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdab4008d30 legacy=0x7fdab4006590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 legacy=0x7fdab40190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab4005e90 legacy=0x7fdab400bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdab4007820 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdab4006ca0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:39.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdab4013080 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4007080 con 0x7fdab4007820 2024-03-25T18:14:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4007ca0 con 0x7fdab4006ca0 2024-03-25T18:14:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4007210 con 0x7fdab4013080 2024-03-25T18:14:39.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:39.335+0000 7fdac4d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182965551 0 0) 0x7fdab4007080 con 0x7fdab4007820 2024-03-25T18:14:40.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:14:40.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.203+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab0003520 con 0x7fdab4007820 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417901165 0 0) 0x7fdab4007210 con 0x7fdab4013080 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:14:40.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.234+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab4007080 con 0x7fdab4013080 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012678271 0 0) 0x7fdab0003520 con 0x7fdab4007820 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6790/0 conn(0x7fdab4013080 legacy=0x7fdab4015470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 legacy=0x7fdab400bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab40073a0 con 0x7fdab4007820 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdac5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdacd04a640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:14:40.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.255+0000 7fdacd04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdab8004010 con 0x7fdab4007820 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: dump: 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdab80042e0 con 0x7fdab4007820 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 1 -- 172.21.15.87:0/3290102026 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdab8004670 con 0x7fdab4007820 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdacd04a640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdacd04a640 4 set_mon_vals no callback set 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.284+0000 7fdac5d9b640 10 monclient: dump: 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:40.287 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 10 monclient: shutdown 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 1 -- 172.21.15.87:0/3290102026 >> v1:172.21.15.87:6789/0 conn(0x7fdab4007820 legacy=0x7fdab400ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 1 -- 172.21.15.87:0/3290102026 shutdown_connections 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 1 -- 172.21.15.87:0/3290102026 >> 172.21.15.87:0/3290102026 conn(0x7fdac80b7f40 msgr2=0x7fdac80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 1 -- 172.21.15.87:0/3290102026 shutdown_connections 2024-03-25T18:14:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.285+0000 7fdacd04a640 1 -- 172.21.15.87:0/3290102026 wait complete. 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 10 monclient: build_initial_monmap 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 10 monclient: monmap: 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 librados: starting msgr at 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 librados: starting objecter 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 Processor -- start 2024-03-25T18:14:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 -- start start 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 librados: setting wanted keys 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 1 librados: calling monclient init 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.286+0000 7fdacd04a640 10 monclient: init 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-b con 0x7fdab4006ca0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-c con 0x7fdab4007820 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient(hunting): picked mon.noname-a con 0x7fdab4013080 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80ad760 con 0x7fdab4013080 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac81d79b0 con 0x7fdab4006ca0 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80ab910 con 0x7fdab4007820 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.287+0000 7fdacd04a640 10 monclient(hunting): authenticate will time out at 7795.582254s 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.288+0000 7fdac659c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fdab4007820 0x7fdac80b1250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:58066/0 (socket says 172.21.15.87:58066) 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.288+0000 7fdac659c640 1 -- 172.21.15.87:0/2980364361 learned_addr learned my addr 172.21.15.87:0/2980364361 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.288+0000 7fdaaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdab4007820 2024-03-25T18:14:40.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.288+0000 7fdaaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.294+0000 7fdaaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdab4006ca0 2024-03-25T18:14:40.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.294+0000 7fdaaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.295+0000 7fdaaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdab4013080 2024-03-25T18:14:40.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.295+0000 7fdaaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773806824 0 0) 0x7fdab4007080 con 0x7fdab4013080 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:14:40.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.344+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab400bc70 con 0x7fdab4013080 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255870431 0 0) 0x7fdab400bc70 con 0x7fdab4013080 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdab4007080 con 0x7fdab4013080 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdac00043f0 con 0x7fdab4013080 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:40.364 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:40.365 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:40.365 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:40.365 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:40.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.363+0000 7fdaaf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634766826 0 0) 0x7fdab40073a0 con 0x7fdab4006ca0 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:14:40.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.366+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab400bc70 con 0x7fdab4006ca0 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469537523 0 0) 0x7fdac80ab910 con 0x7fdab4007820 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:14:40.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.371+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab40073a0 con 0x7fdab4007820 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655327312 0 0) 0x7fdab4007080 con 0x7fdab4013080 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 >> v1:172.21.15.87:6790/0 conn(0x7fdab4007820 legacy=0x7fdac80b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 >> v1:172.21.15.171:6789/0 conn(0x7fdab4006ca0 legacy=0x7fdac81d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 1 monclient: found mon.a 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac81d7610 con 0x7fdab4013080 2024-03-25T18:14:40.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdaaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 10 monclient: _renew_subs 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac80ad760 con 0x7fdab4013080 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 10 monclient: _renew_subs 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac80ad760 con 0x7fdab4013080 2024-03-25T18:14:40.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.383+0000 7fdacd04a640 1 librados: init done 2024-03-25T18:14:40.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.384+0000 7fdacd04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.384+0000 7fdacd04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.384+0000 7fdacd04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.384+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac80ad760 con 0x7fdab4013080 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac00045c0 con 0x7fdab4013080 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdac00049d0 con 0x7fdab4013080 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: dump: 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:40.565 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdac0004f80 con 0x7fdab4013080 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4021763483 0 0) 0x7fdac0006730 con 0x7fdab4013080 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 client.4107.objecter ms_dispatch 0x7fdac8000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdac559a640 4 set_mon_vals no callback set 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:14:40.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdac0006b50 con 0x7fdab4013080 2024-03-25T18:14:40.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.564+0000 7fdaaf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:40.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.634+0000 7fdacd04a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:40.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.634+0000 7fdacd04a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:40.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.634+0000 7fdacd04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:40.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.634+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdac80ad760 con 0x7fdab4013080 2024-03-25T18:14:40.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdaaf7fe640 1 -- 172.21.15.87:0/2980364361 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fdac0005990 con 0x7fdab4013080 2024-03-25T18:14:40.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdaaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:40.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdaaf7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:14:40.801 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdacd04a640 10 librados: watch_flush enter 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdacd04a640 10 librados: watch_flush exit 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdacd04a640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.800+0000 7fdacd04a640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 10 monclient: shutdown 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 >> v1:172.21.15.87:6789/0 conn(0x7fdab4013080 legacy=0x7fdac81e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 shutdown_connections 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 >> 172.21.15.87:0/2980364361 conn(0x7fdac80b7f40 msgr2=0x7fdac8139460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 shutdown_connections 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 -- 172.21.15.87:0/2980364361 wait complete. 2024-03-25T18:14:40.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:40.801+0000 7fdacd04a640 1 librados: shutdown 2024-03-25T18:14:40.813 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:14:40.813 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:14:40.813 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:40.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:14:40.841 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:40.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:14:40.910 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:40.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:14:40.978 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-25T18:14:40.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:14:41.045 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:41.045 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:14:41.073 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:41.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:14:41.093 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.092+0000 7fd38a82e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:41.119 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:41.119 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:14:41.149 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.149+0000 7fd38a82e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:41.176 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-03-25T18:14:41.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:14:41.205 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.205+0000 7fdf7be21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:41.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 980ca77f-5be5-4342-81e2-07ef296105f3 0 2024-03-25T18:14:41.273 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.273+0000 7fdf7be21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.394+0000 7f4e574ba640 10 monclient: get_monmap_and_config 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.394+0000 7f4e574ba640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.394+0000 7f4e574ba640 10 monclient: monmap: 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.395 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:41.395775+0000 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:41.395775+0000 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:41.396 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.395+0000 7f4e574ba640 1 Processor -- start 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.395+0000 7f4e574ba640 1 -- start start 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.395+0000 7f4e574ba640 10 monclient: init 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:41.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500b4700 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4e50142a20 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4e500a96b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500a9d60 con 0x7f4e50142a20 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5013ff10 con 0x7f4e500a96b0 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500aa1f0 con 0x7f4e500b4700 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.396+0000 7f4e574ba640 10 monclient(hunting): authenticate will time out at 7785.537026s 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.397+0000 7f4e54a55640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4e50142a20 0x7f4e5013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.171:56262/0 (socket says 172.21.15.171:56262) 2024-03-25T18:14:41.398 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.397+0000 7f4e54a55640 1 -- 172.21.15.171:0/2377309432 learned_addr learned my addr 172.21.15.171:0/2377309432 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899033471 0 0) 0x7f4e500a9d60 con 0x7f4e50142a20 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:14:41.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.399+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3c003520 con 0x7f4e50142a20 2024-03-25T18:14:41.403 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034423677 0 0) 0x7f4e3c003520 con 0x7f4e50142a20 2024-03-25T18:14:41.403 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 >> v1:172.21.15.87:6790/0 conn(0x7f4e500b4700 legacy=0x7f4e50142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 >> v1:172.21.15.171:6789/0 conn(0x7f4e500a96b0 legacy=0x7f4e5013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 monclient: found mon.noname-a 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500b4ae0 con 0x7f4e50142a20 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e574ba640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e40001690 con 0x7f4e50142a20 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e574ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:41.404 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e47fff640 10 monclient: dump: 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.405 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.403+0000 7f4e574ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e40001860 con 0x7f4e50142a20 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 1 -- 172.21.15.171:0/2377309432 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e40001cb0 con 0x7f4e50142a20 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e574ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e574ba640 4 set_mon_vals no callback set 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.404+0000 7f4e47fff640 10 monclient: dump: 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.407 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 10 monclient: shutdown 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 1 -- 172.21.15.171:0/2377309432 >> v1:172.21.15.87:6789/0 conn(0x7f4e50142a20 legacy=0x7f4e5013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 1 -- 172.21.15.171:0/2377309432 shutdown_connections 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 1 -- 172.21.15.171:0/2377309432 >> 172.21.15.171:0/2377309432 conn(0x7f4e500668e0 msgr2=0x7f4e50066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 1 -- 172.21.15.171:0/2377309432 shutdown_connections 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.405+0000 7f4e574ba640 1 -- 172.21.15.171:0/2377309432 wait complete. 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 10 monclient: monmap: 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:41.408 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 librados: starting msgr at 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 librados: starting objecter 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 Processor -- start 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 -- start start 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 librados: setting wanted keys 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 1 librados: calling monclient init 2024-03-25T18:14:41.409 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.406+0000 7f4e574ba640 10 monclient: init 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a96b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4e50142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5013ac80 con 0x7f4e500a96b0 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5013bec0 con 0x7f4e500b4700 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50138620 con 0x7f4e50142a20 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.411 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e574ba640 10 monclient(hunting): authenticate will time out at 7785.547882s 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e55256640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4e500a96b0 0x7f4e50137340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.171:56264/0 (socket says 172.21.15.171:56264) 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.407+0000 7f4e55256640 1 -- 172.21.15.171:0/2221387549 learned_addr learned my addr 172.21.15.171:0/2221387549 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.408+0000 7f4e45ffb640 10 client.?.objecter ms_handle_connect 0x7f4e500a96b0 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.408+0000 7f4e45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232367408 0 0) 0x7f4e5013ac80 con 0x7f4e500a96b0 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:14:41.412 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.409+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e34004eb0 con 0x7f4e500a96b0 2024-03-25T18:14:41.413 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.413+0000 7f4e45ffb640 10 client.?.objecter ms_handle_connect 0x7f4e500b4700 2024-03-25T18:14:41.414 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.413+0000 7f4e45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3726419584 0 0) 0x7f4e34004eb0 con 0x7f4e500a96b0 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5013ac80 con 0x7f4e500a96b0 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e4c001960 con 0x7f4e500a96b0 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:41.416 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.415+0000 7f4e45ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:41.419 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.418+0000 7f4e45ffb640 10 client.?.objecter ms_handle_connect 0x7f4e50142a20 2024-03-25T18:14:41.419 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.418+0000 7f4e45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3810763479 0 0) 0x7f4e5013ac80 con 0x7f4e500a96b0 2024-03-25T18:14:41.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 >> v1:172.21.15.87:6790/0 conn(0x7f4e50142a20 legacy=0x7f4e5013b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 >> v1:172.21.15.171:6789/0 conn(0x7f4e500b4700 legacy=0x7f4e5013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 1 monclient: found mon.a 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e501387b0 con 0x7f4e500a96b0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e574ba640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e574ba640 10 monclient: _renew_subs 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e574ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.423+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5013cb70 con 0x7f4e500a96b0 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.424+0000 7f4e574ba640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.424+0000 7f4e574ba640 10 monclient: _renew_subs 2024-03-25T18:14:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.424+0000 7f4e574ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.424+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e50137c30 con 0x7f4e500a96b0 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.424+0000 7f4e574ba640 1 librados: init done 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.425+0000 7f4e574ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.425+0000 7f4e574ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.425+0000 7f4e574ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.426 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.425+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e500b1c20 con 0x7f4e500a96b0 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e4c001cc0 con 0x7f4e500a96b0 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e4c0038a0 con 0x7f4e500a96b0 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 10 monclient: dump: 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.445 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4e4c003e50 con 0x7f4e500a96b0 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2383641101 0 0) 0x7f4e4c006390 con 0x7f4e500a96b0 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 10 client.4119.objecter ms_dispatch 0x7f4e50000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e477fe640 4 set_mon_vals no callback set 2024-03-25T18:14:41.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.444+0000 7f4e45ffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:14:41.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.564+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4e4c004100 con 0x7f4e500a96b0 2024-03-25T18:14:41.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.564+0000 7f4e45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.564+0000 7f4e45ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:41.637 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.636+0000 7f4e574ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "980ca77f-5be5-4342-81e2-07ef296105f3", "id": 0}] 2024-03-25T18:14:41.637 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.636+0000 7f4e574ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "980ca77f-5be5-4342-81e2-07ef296105f3", "id": 0}] 2024-03-25T18:14:41.637 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.636+0000 7f4e574ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.637 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.636+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd new", "uuid": "980ca77f-5be5-4342-81e2-07ef296105f3", "id": 0} v 0) v1 -- 0x7f4e5004f730 con 0x7f4e500a96b0 2024-03-25T18:14:41.710 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.710+0000 7fd38a82e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:41.721 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.720+0000 7f4e45ffb640 1 -- 172.21.15.171:0/2221387549 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "980ca77f-5be5-4342-81e2-07ef296105f3", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2612682076 0 732707515) 0x7f4e4c004450 con 0x7f4e500a96b0 2024-03-25T18:14:41.721 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.720+0000 7f4e45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "980ca77f-5be5-4342-81e2-07ef296105f3", "id": 0}] 2024-03-25T18:14:41.721 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.720+0000 7f4e45ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:41.721 INFO:teuthology.orchestra.run.smithi171.stdout:0 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 10 librados: watch_flush enter 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 10 librados: watch_flush exit 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 10 monclient: shutdown 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.721+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 >> v1:172.21.15.87:6789/0 conn(0x7f4e500a96b0 legacy=0x7f4e50137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.722+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 shutdown_connections 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.722+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 >> 172.21.15.171:0/2221387549 conn(0x7f4e500668e0 msgr2=0x7f4e5013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.722+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 shutdown_connections 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.722+0000 7f4e574ba640 1 -- 172.21.15.171:0/2221387549 wait complete. 2024-03-25T18:14:41.723 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.722+0000 7f4e574ba640 1 librados: shutdown 2024-03-25T18:14:41.736 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new db380654-84f4-4fd2-b2a0-d94129e518b3 1 2024-03-25T18:14:41.769 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.769+0000 7fd38a82e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:41.816 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.816+0000 7fd38a82e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:41.822 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.821+0000 7fdf7be21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.836+0000 7f7037ac1640 10 monclient: get_monmap_and_config 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.836+0000 7f7037ac1640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.837+0000 7f7037ac1640 10 monclient: monmap: 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.837 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:41.837943+0000 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:41.837943+0000 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:41.838 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.837+0000 7f7037ac1640 1 Processor -- start 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.837+0000 7f7037ac1640 1 -- start start 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.837+0000 7f7037ac1640 10 monclient: init 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:41.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f70300b4700 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f7030142a20 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f70300a96b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300a9d60 con 0x7f7030142a20 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703013ff10 con 0x7f70300a96b0 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300aa1f0 con 0x7f70300b4700 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f7037ac1640 10 monclient(hunting): authenticate will time out at 7785.979188s 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f703605e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f70300a96b0 0x7f703013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52652/0 (socket says 172.21.15.171:52652) 2024-03-25T18:14:41.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.838+0000 7f703605e640 1 -- 172.21.15.171:0/1040080828 learned_addr learned my addr 172.21.15.171:0/1040080828 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214402006 0 0) 0x7f703013ff10 con 0x7f70300a96b0 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c003520 con 0x7f70300a96b0 2024-03-25T18:14:41.842 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283787351 0 0) 0x7f70300a9d60 con 0x7f7030142a20 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f703013ff10 con 0x7f7030142a20 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179928196 0 0) 0x7f70300aa1f0 con 0x7f70300b4700 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:14:41.843 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.841+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70300a9d60 con 0x7f70300b4700 2024-03-25T18:14:41.845 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354283098 0 0) 0x7f703013ff10 con 0x7f7030142a20 2024-03-25T18:14:41.845 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 >> v1:172.21.15.87:6790/0 conn(0x7f70300b4700 legacy=0x7f7030142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.845 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 >> v1:172.21.15.171:6789/0 conn(0x7f70300a96b0 legacy=0x7f703013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 monclient: found mon.noname-a 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70300b4ae0 con 0x7f7030142a20 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f702c0015f0 con 0x7f7030142a20 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f703485b640 10 monclient: dump: 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.846 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f7037ac1640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:14:41.847 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.844+0000 7f7037ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f702c001750 con 0x7f7030142a20 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 1 -- 172.21.15.171:0/1040080828 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f702c001b60 con 0x7f7030142a20 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f703485b640 10 monclient: dump: 2024-03-25T18:14:41.848 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f7037ac1640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.846+0000 7f7037ac1640 4 set_mon_vals no callback set 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 10 monclient: shutdown 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 1 -- 172.21.15.171:0/1040080828 >> v1:172.21.15.87:6789/0 conn(0x7f7030142a20 legacy=0x7f703013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 1 -- 172.21.15.171:0/1040080828 shutdown_connections 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 1 -- 172.21.15.171:0/1040080828 >> 172.21.15.171:0/1040080828 conn(0x7f70300668e0 msgr2=0x7f7030066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 1 -- 172.21.15.171:0/1040080828 shutdown_connections 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.847+0000 7f7037ac1640 1 -- 172.21.15.171:0/1040080828 wait complete. 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 10 monclient: monmap: 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:41.850 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 librados: starting msgr at 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 librados: starting objecter 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 Processor -- start 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 -- start start 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 librados: setting wanted keys 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 1 librados: calling monclient init 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 10 monclient: init 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.848+0000 7f7037ac1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f70300a96b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f70300b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f7030142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703013c060 con 0x7f70300a96b0 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703013ca90 con 0x7f70300b4700 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300aba90 con 0x7f7030142a20 2024-03-25T18:14:41.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f7037ac1640 10 monclient(hunting): authenticate will time out at 7785.989824s 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f703585d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f70300a96b0 0x7f703013b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.171:56284/0 (socket says 172.21.15.171:56284) 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f703585d640 1 -- 172.21.15.171:0/1226948519 learned_addr learned my addr 172.21.15.171:0/1226948519 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f70227fc640 10 client.?.objecter ms_handle_connect 0x7f70300a96b0 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.849+0000 7f70227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.850+0000 7f70227fc640 10 client.?.objecter ms_handle_connect 0x7f7030142a20 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.850+0000 7f70227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 client.?.objecter ms_handle_connect 0x7f70300b4700 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738214355 0 0) 0x7f703013c060 con 0x7f70300a96b0 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.851+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7014004eb0 con 0x7f70300a96b0 2024-03-25T18:14:41.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291041404 0 0) 0x7f70300aba90 con 0x7f7030142a20 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.852+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f703013c060 con 0x7f7030142a20 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754388060 0 0) 0x7f703013ca90 con 0x7f70300b4700 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:14:41.856 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.853+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70300aba90 con 0x7f70300b4700 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2750043847 0 0) 0x7f7014004eb0 con 0x7f70300a96b0 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f703013ca90 con 0x7f70300a96b0 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7024001a10 con 0x7f70300a96b0 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 10 monclient(hunting): dump: 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.857 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425040901 0 0) 0x7f703013c060 con 0x7f7030142a20 2024-03-25T18:14:41.858 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.856+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7014004eb0 con 0x7f7030142a20 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.857+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7028001b90 con 0x7f7030142a20 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.857+0000 7f70227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.857+0000 7f70227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.857+0000 7f70227fc640 10 monclient(hunting): dump: 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.859 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.857+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345884229 0 0) 0x7f70300aba90 con 0x7f70300b4700 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.858+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f703013c060 con 0x7f70300b4700 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.858+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f702c001a60 con 0x7f70300b4700 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.858+0000 7f70227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.858+0000 7f70227fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.858+0000 7f70227fc640 10 monclient(hunting): dump: 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.860 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.861 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.861 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.861 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1934570424 0 0) 0x7f703013ca90 con 0x7f70300a96b0 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 >> v1:172.21.15.87:6790/0 conn(0x7f7030142a20 legacy=0x7f7030138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 >> v1:172.21.15.171:6789/0 conn(0x7f70300b4700 legacy=0x7f7030138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 1 monclient: found mon.a 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703013af40 con 0x7f70300a96b0 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f70227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.862 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f7037ac1640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f7037ac1640 10 monclient: _renew_subs 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.859+0000 7f7037ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703013b1f0 con 0x7f70300a96b0 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 monclient: _renew_subs 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70301e2e40 con 0x7f70300a96b0 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 1 librados: init done 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:41.863 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.860+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703004f730 con 0x7f70300a96b0 2024-03-25T18:14:41.864 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7024001c70 con 0x7f70300a96b0 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7024003a60 con 0x7f70300a96b0 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 10 monclient: dump: 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:41.865 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7024004010 con 0x7f70300a96b0 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f7023fff640 4 set_mon_vals no callback set 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2467411996 0 0) 0x7f7024006670 con 0x7f70300a96b0 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 10 client.4125.objecter ms_dispatch 0x7f7030000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:14:41.866 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.864+0000 7f70227fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:14:41.874 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.874+0000 7fd38a82e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:41.878 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.878+0000 7fdf7be21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:41.917 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:41.917+0000 7fd38a82e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:41.923 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.923+0000 7fdf7be21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:41.957 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.956+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f70240042c0 con 0x7f70300a96b0 2024-03-25T18:14:41.957 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.956+0000 7f70227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.957 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:41.956+0000 7f70227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:41.979 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:41.978+0000 7fdf7be21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:42.021 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.021+0000 7fdf7be21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:42.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.028+0000 7f7037ac1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "db380654-84f4-4fd2-b2a0-d94129e518b3", "id": 1}] 2024-03-25T18:14:42.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.028+0000 7f7037ac1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "db380654-84f4-4fd2-b2a0-d94129e518b3", "id": 1}] 2024-03-25T18:14:42.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.028+0000 7f7037ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.028+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd new", "uuid": "db380654-84f4-4fd2-b2a0-d94129e518b3", "id": 1} v 0) v1 -- 0x7f7030069f30 con 0x7f70300a96b0 2024-03-25T18:14:42.037 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.036+0000 7fd38a82e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:42.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.066+0000 7f70227fc640 1 -- 172.21.15.171:0/1226948519 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "db380654-84f4-4fd2-b2a0-d94129e518b3", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 113642557 0 940483276) 0x7f7024006930 con 0x7f70300a96b0 2024-03-25T18:14:42.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.066+0000 7f70227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "db380654-84f4-4fd2-b2a0-d94129e518b3", "id": 1}] 2024-03-25T18:14:42.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.066+0000 7f70227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:42.067 INFO:teuthology.orchestra.run.smithi171.stdout:1 2024-03-25T18:14:42.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 10 librados: watch_flush enter 2024-03-25T18:14:42.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 10 librados: watch_flush exit 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 10 monclient: shutdown 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 >> v1:172.21.15.87:6789/0 conn(0x7f70300a96b0 legacy=0x7f703013b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 shutdown_connections 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 >> 172.21.15.171:0/1226948519 conn(0x7f70300668e0 msgr2=0x7f70300b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 shutdown_connections 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 -- 172.21.15.171:0/1226948519 wait complete. 2024-03-25T18:14:42.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.067+0000 7f7037ac1640 1 librados: shutdown 2024-03-25T18:14:42.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new c31c9f22-b8b1-4060-bbfc-18fdf310a080 2 2024-03-25T18:14:42.111 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.111+0000 7fd38a82e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:42.143 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.142+0000 7fdf7be21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:42.160 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.160+0000 7fd38a82e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:42.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:42.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 10 monclient: monmap: 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:42.212095+0000 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:42.212095+0000 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:42.212 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.213 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.212+0000 7fdf7be21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 1 Processor -- start 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 1 -- start start 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.211+0000 7f42c360d640 10 monclient: init 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc070210 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0b06d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc0705f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b0b80 con 0x7f42bc0705f0 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b0430 con 0x7f42bc070210 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc071670 con 0x7f42bc0b06d0 2024-03-25T18:14:42.214 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c360d640 10 monclient(hunting): authenticate will time out at 7786.353238s 2024-03-25T18:14:42.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c13a9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42bc070210 0x7f42bc140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52666/0 (socket says 172.21.15.171:52666) 2024-03-25T18:14:42.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.212+0000 7f42c13a9640 1 -- 172.21.15.171:0/2283404322 learned_addr learned my addr 172.21.15.171:0/2283404322 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.215 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.215+0000 7fd38a82e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:42.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037530587 0 0) 0x7f42b8001110 con 0x7f42bc070210 2024-03-25T18:14:42.215 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.215+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a8003520 con 0x7f42bc070210 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539481749 0 0) 0x7f42bc0b0b80 con 0x7f42bc0705f0 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.216 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b8001110 con 0x7f42bc0705f0 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564273038 0 0) 0x7f42bc071670 con 0x7f42bc0b06d0 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:14:42.217 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.216+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc0b0b80 con 0x7f42bc0b06d0 2024-03-25T18:14:42.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870814807 0 0) 0x7f42a8003520 con 0x7f42bc070210 2024-03-25T18:14:42.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 >> v1:172.21.15.87:6790/0 conn(0x7f42bc0b06d0 legacy=0x7f42bc0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 >> v1:172.21.15.87:6789/0 conn(0x7f42bc0705f0 legacy=0x7f42bc14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.219 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 monclient: found mon.noname-c 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc070a60 con 0x7f42bc070210 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42b8001680 con 0x7f42bc070210 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42b3fff640 10 monclient: dump: 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.220 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42c360d640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:14:42.221 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.218+0000 7f42c360d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.244 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b80017e0 con 0x7f42bc070210 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 1 -- 172.21.15.171:0/2283404322 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42b8001b70 con 0x7f42bc070210 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42c360d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42c360d640 4 set_mon_vals no callback set 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42b3fff640 10 monclient: dump: 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.245 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.246 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.248 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42c360d640 10 monclient: shutdown 2024-03-25T18:14:42.248 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42c360d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.248 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.244+0000 7f42c360d640 1 -- 172.21.15.171:0/2283404322 >> v1:172.21.15.171:6789/0 conn(0x7f42bc070210 legacy=0x7f42bc140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 1 -- 172.21.15.171:0/2283404322 shutdown_connections 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 1 -- 172.21.15.171:0/2283404322 >> 172.21.15.171:0/2283404322 conn(0x7f42bc0b5cd0 msgr2=0x7f42bc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 1 -- 172.21.15.171:0/2283404322 shutdown_connections 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 1 -- 172.21.15.171:0/2283404322 wait complete. 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.245+0000 7f42c360d640 10 monclient: monmap: 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.249 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 librados: starting msgr at 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 librados: starting objecter 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 Processor -- start 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 -- start start 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 librados: setting wanted keys 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 librados: calling monclient init 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient: init 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc070210 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0705f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc0b06d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc06dc60 con 0x7f42bc070210 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc06d980 con 0x7f42bc0705f0 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0ae680 con 0x7f42bc0b06d0 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.250 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.246+0000 7f42c360d640 10 monclient(hunting): authenticate will time out at 7786.387611s 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42c0ba8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42bc0705f0 0x7f42bc131e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52678/0 (socket says 172.21.15.171:52678) 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42c0ba8640 1 -- 172.21.15.171:0/3669450845 learned_addr learned my addr 172.21.15.171:0/3669450845 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter ms_handle_connect 0x7f42bc0705f0 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter ms_handle_connect 0x7f42bc070210 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter ms_handle_connect 0x7f42bc0b06d0 2024-03-25T18:14:42.251 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.247+0000 7f42b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366147601 0 0) 0x7f42bc06d980 con 0x7f42bc0705f0 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42bc0705f0 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679317473 0 0) 0x7f42bc06dc60 con 0x7f42bc070210 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:14:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc06d980 con 0x7f42bc070210 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841731526 0 0) 0x7f42bc0ae680 con 0x7f42bc0b06d0 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:14:42.253 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.249+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc06dc60 con 0x7f42bc0b06d0 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 790193728 0 0) 0x7f42a0004eb0 con 0x7f42bc0705f0 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42bc0ae680 con 0x7f42bc0705f0 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42ac001970 con 0x7f42bc0705f0 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 469654957 0 0) 0x7f42bc06d980 con 0x7f42bc070210 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42bc070210 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42b80012a0 con 0x7f42bc070210 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2855391311 0 0) 0x7f42bc06dc60 con 0x7f42bc0b06d0 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42bc06d980 con 0x7f42bc0b06d0 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42b4001cb0 con 0x7f42bc0b06d0 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.254+0000 7f42b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.258 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259879321 0 0) 0x7f42a0004eb0 con 0x7f42bc070210 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 >> v1:172.21.15.87:6790/0 conn(0x7f42bc0b06d0 legacy=0x7f42bc1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 >> v1:172.21.15.171:6789/0 conn(0x7f42bc0705f0 legacy=0x7f42bc131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 1 monclient: found mon.a 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0af2e0 con 0x7f42bc070210 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 10 monclient: _renew_subs 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc06cd30 con 0x7f42bc070210 2024-03-25T18:14:42.260 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 10 monclient: _renew_subs 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc1daa70 con 0x7f42bc070210 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.257+0000 7f42c360d640 1 librados: init done 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.258+0000 7f42c360d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.258+0000 7f42c360d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.258+0000 7f42c360d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.261 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.258+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc04f730 con 0x7f42bc070210 2024-03-25T18:14:42.261 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.261+0000 7fdf7be21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:42.266 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.265+0000 7fd38a82e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:42.315 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.315+0000 7fdf7be21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b8004a90 con 0x7f42bc070210 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f42b8004da0 con 0x7f42bc070210 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 10 monclient: dump: 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.355 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f42b8005290 con 0x7f42bc070210 2024-03-25T18:14:42.356 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.354+0000 7f42b37fe640 4 set_mon_vals no callback set 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 683879444 0 0) 0x7f42b8006730 con 0x7f42bc070210 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 10 client.4131.objecter ms_dispatch 0x7f42bc000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f42b8006310 con 0x7f42bc070210 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.355+0000 7f42b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:42.364 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.363+0000 7fdf7be21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:42.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.426+0000 7f42c360d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c31c9f22-b8b1-4060-bbfc-18fdf310a080", "id": 2}] 2024-03-25T18:14:42.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.426+0000 7f42c360d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c31c9f22-b8b1-4060-bbfc-18fdf310a080", "id": 2}] 2024-03-25T18:14:42.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.426+0000 7f42c360d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.426+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c31c9f22-b8b1-4060-bbfc-18fdf310a080", "id": 2} v 0) v1 -- 0x7f42bc0ad8b0 con 0x7f42bc070210 2024-03-25T18:14:42.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.454+0000 7f42b1ffb640 1 -- 172.21.15.171:0/3669450845 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c31c9f22-b8b1-4060-bbfc-18fdf310a080", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3369953372 0 216599125) 0x7f42b8006530 con 0x7f42bc070210 2024-03-25T18:14:42.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.454+0000 7f42b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c31c9f22-b8b1-4060-bbfc-18fdf310a080", "id": 2}] 2024-03-25T18:14:42.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.454+0000 7f42b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:42.455 INFO:teuthology.orchestra.run.smithi171.stdout:2 2024-03-25T18:14:42.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 10 librados: watch_flush enter 2024-03-25T18:14:42.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 10 librados: watch_flush exit 2024-03-25T18:14:42.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:14:42.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 10 monclient: shutdown 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 >> v1:172.21.15.87:6789/0 conn(0x7f42bc070210 legacy=0x7f42bc0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 shutdown_connections 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 >> 172.21.15.171:0/3669450845 conn(0x7f42bc0b5cd0 msgr2=0x7f42bc0afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 shutdown_connections 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 -- 172.21.15.171:0/3669450845 wait complete. 2024-03-25T18:14:42.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.455+0000 7f42c360d640 1 librados: shutdown 2024-03-25T18:14:42.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 18f5b86f-145b-4d76-b45d-f921a4b316df 3 2024-03-25T18:14:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.571+0000 7f6639445640 10 monclient: get_monmap_and_config 2024-03-25T18:14:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.571+0000 7f6639445640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.572+0000 7f6639445640 10 monclient: monmap: 2024-03-25T18:14:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:42.572942+0000 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:42.572942+0000 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:42.573 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.572+0000 7f6639445640 1 Processor -- start 2024-03-25T18:14:42.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.572+0000 7f6639445640 1 -- start start 2024-03-25T18:14:42.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.572+0000 7f6639445640 10 monclient: init 2024-03-25T18:14:42.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient(hunting): picked mon.noname-c con 0x7f66340b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient(hunting): picked mon.noname-b con 0x7f6634142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient(hunting): picked mon.noname-a con 0x7f66340a96b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340a9d60 con 0x7f66340a96b0 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663413ff10 con 0x7f66340b4700 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340aa1f0 con 0x7f6634142a20 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f6639445640 10 monclient(hunting): authenticate will time out at 7786.713960s 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f66337fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f66340a96b0 0x7f663413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.171:56300/0 (socket says 172.21.15.171:56300) 2024-03-25T18:14:42.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.573+0000 7f66337fe640 1 -- 172.21.15.171:0/1711415745 learned_addr learned my addr 172.21.15.171:0/1711415745 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699209930 0 0) 0x7f66340a9d60 con 0x7f66340a96b0 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c003520 con 0x7f66340a96b0 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912603608 0 0) 0x7f66340aa1f0 con 0x7f6634142a20 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66340a9d60 con 0x7f6634142a20 2024-03-25T18:14:42.577 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984379077 0 0) 0x7f663413ff10 con 0x7f66340b4700 2024-03-25T18:14:42.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:14:42.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.576+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66340aa1f0 con 0x7f66340b4700 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4193630291 0 0) 0x7f661c003520 con 0x7f66340a96b0 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 >> v1:172.21.15.87:6790/0 conn(0x7f6634142a20 legacy=0x7f663413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 >> v1:172.21.15.171:6789/0 conn(0x7f66340b4700 legacy=0x7f6634142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 monclient: found mon.noname-a 2024-03-25T18:14:42.580 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66340b4ae0 con 0x7f66340a96b0 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6624001690 con 0x7f66340a96b0 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6639445640 5 monclient: authenticate success, global_id 4134 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6639445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6631ffb640 10 monclient: dump: 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.581 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.579+0000 7f6639445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66240017f0 con 0x7f66340a96b0 2024-03-25T18:14:42.584 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6639445640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6639445640 4 set_mon_vals no callback set 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 1 -- 172.21.15.171:0/1711415745 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6624001c00 con 0x7f66340a96b0 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6631ffb640 10 monclient: dump: 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.585 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6639445640 10 monclient: shutdown 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6639445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.581+0000 7f6639445640 1 -- 172.21.15.171:0/1711415745 >> v1:172.21.15.87:6789/0 conn(0x7f66340a96b0 legacy=0x7f663413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 -- 172.21.15.171:0/1711415745 shutdown_connections 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 -- 172.21.15.171:0/1711415745 >> 172.21.15.171:0/1711415745 conn(0x7f66340668e0 msgr2=0x7f6634066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 -- 172.21.15.171:0/1711415745 shutdown_connections 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 -- 172.21.15.171:0/1711415745 wait complete. 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 10 monclient: monmap: 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:42.586 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 librados: starting msgr at 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.582+0000 7f6639445640 1 librados: starting objecter 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 Processor -- start 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 -- start start 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 librados: setting wanted keys 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 librados: calling monclient init 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient: init 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient(hunting): picked mon.noname-c con 0x7f66340a96b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient(hunting): picked mon.noname-b con 0x7f66340b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient(hunting): picked mon.noname-a con 0x7f6634142a20 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340acdb0 con 0x7f6634142a20 2024-03-25T18:14:42.587 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634133a20 con 0x7f66340b4700 2024-03-25T18:14:42.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340ac280 con 0x7f66340a96b0 2024-03-25T18:14:42.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.583+0000 7f6639445640 10 monclient(hunting): authenticate will time out at 7786.724580s 2024-03-25T18:14:42.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f6632ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f66340a96b0 0x7f66340ac6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.171:38584/0 (socket says 172.21.15.171:38584) 2024-03-25T18:14:42.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f6632ffd640 1 -- 172.21.15.171:0/4030281116 learned_addr learned my addr 172.21.15.171:0/4030281116 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter ms_handle_connect 0x7f6634142a20 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter ms_handle_connect 0x7f66340b4700 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter ms_handle_connect 0x7f66340a96b0 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.584+0000 7f661b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373249253 0 0) 0x7f6634133a20 con 0x7f66340b4700 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:14:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620004eb0 con 0x7f66340b4700 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.586+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741484328 0 0) 0x7f66340ac280 con 0x7f66340a96b0 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634133a20 con 0x7f66340a96b0 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586733324 0 0) 0x7f66340acdb0 con 0x7f6634142a20 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:14:42.590 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.587+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66340ac280 con 0x7f6634142a20 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286964419 0 0) 0x7f66340ac280 con 0x7f6634142a20 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66340acdb0 con 0x7f6634142a20 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6624001b00 con 0x7f6634142a20 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.593 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190743569 0 0) 0x7f6634133a20 con 0x7f66340a96b0 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66340ac280 con 0x7f66340a96b0 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f662c001d70 con 0x7f66340a96b0 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.594 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.591+0000 7f661b7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309028056 0 0) 0x7f6620004eb0 con 0x7f66340b4700 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6634133a20 con 0x7f66340b4700 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6628001910 con 0x7f66340b4700 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.592+0000 7f661b7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.596 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3797286901 0 0) 0x7f66340acdb0 con 0x7f6634142a20 2024-03-25T18:14:42.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 >> v1:172.21.15.87:6790/0 conn(0x7f66340a96b0 legacy=0x7f66340ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 >> v1:172.21.15.171:6789/0 conn(0x7f66340b4700 legacy=0x7f6634132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 1 monclient: found mon.a 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663413b890 con 0x7f6634142a20 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f661b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 10 monclient: _renew_subs 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663413cc20 con 0x7f6634142a20 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 10 monclient: _renew_subs 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f663413b3c0 con 0x7f6634142a20 2024-03-25T18:14:42.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.595+0000 7f6639445640 1 librados: init done 2024-03-25T18:14:42.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.596+0000 7f6639445640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.596+0000 7f6639445640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.596+0000 7f6639445640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.596+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663404f730 con 0x7f6634142a20 2024-03-25T18:14:42.665 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.664+0000 7fd38a82e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66240010e0 con 0x7f6634142a20 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6624000f60 con 0x7f6634142a20 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 10 monclient: dump: 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.693 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6624004fc0 con 0x7f6634142a20 2024-03-25T18:14:42.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f66317fa640 4 set_mon_vals no callback set 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2608063387 0 0) 0x7f6624006670 con 0x7f6634142a20 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 10 client.4137.objecter ms_dispatch 0x7f6634000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.692+0000 7f661b7fe640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.693+0000 7f661b7fe640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.693+0000 7f661b7fe640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.693+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6624006960 con 0x7f6634142a20 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.693+0000 7f661b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.693+0000 7f661b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:42.757 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.756+0000 7fdf7be21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:42.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.764+0000 7f6639445640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "18f5b86f-145b-4d76-b45d-f921a4b316df", "id": 3}] 2024-03-25T18:14:42.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.765+0000 7f6639445640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "18f5b86f-145b-4d76-b45d-f921a4b316df", "id": 3}] 2024-03-25T18:14:42.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.765+0000 7f6639445640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:42.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.765+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd new", "uuid": "18f5b86f-145b-4d76-b45d-f921a4b316df", "id": 3} v 0) v1 -- 0x7f66340b7680 con 0x7f6634142a20 2024-03-25T18:14:42.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.796+0000 7f661b7fe640 1 -- 172.21.15.171:0/4030281116 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18f5b86f-145b-4d76-b45d-f921a4b316df", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3959829288 0 525046306) 0x7f66240062b0 con 0x7f6634142a20 2024-03-25T18:14:42.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.796+0000 7f661b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "18f5b86f-145b-4d76-b45d-f921a4b316df", "id": 3}] 2024-03-25T18:14:42.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.796+0000 7f661b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:42.797 INFO:teuthology.orchestra.run.smithi171.stdout:3 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 10 librados: watch_flush enter 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 10 librados: watch_flush exit 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 10 monclient: shutdown 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 >> v1:172.21.15.87:6789/0 conn(0x7f6634142a20 legacy=0x7f6634133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 shutdown_connections 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 >> 172.21.15.171:0/4030281116 conn(0x7f66340668e0 msgr2=0x7f663413e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 shutdown_connections 2024-03-25T18:14:42.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 -- 172.21.15.171:0/4030281116 wait complete. 2024-03-25T18:14:42.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.798+0000 7f6639445640 1 librados: shutdown 2024-03-25T18:14:42.805 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.805+0000 7fd38a82e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:42.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new a6cae3ee-630b-444f-84bd-bd172830fcab 4 2024-03-25T18:14:42.897 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:42.896+0000 7fdf7be21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 10 monclient: get_monmap_and_config 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 10 monclient: monmap: 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:42.914287+0000 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:42.914287+0000 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:42.914 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:42.915 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 1 Processor -- start 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 1 -- start start 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.913+0000 7fd334455640 10 monclient: init 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c0b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c142a20 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c0a96b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0a9d60 con 0x7fd32c142a20 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c13ff10 con 0x7fd32c0b4700 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0aa1f0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.914+0000 7fd334455640 10 monclient(hunting): authenticate will time out at 7787.055457s 2024-03-25T18:14:42.916 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.915+0000 7fd3321f1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd32c0b4700 0x7fd32c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52700/0 (socket says 172.21.15.171:52700) 2024-03-25T18:14:42.917 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.915+0000 7fd3321f1640 1 -- 172.21.15.171:0/2737646164 learned_addr learned my addr 172.21.15.171:0/2737646164 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053454447 0 0) 0x7fd32c13ff10 con 0x7fd32c0b4700 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:14:42.918 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.917+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd318003520 con 0x7fd32c0b4700 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634124915 0 0) 0x7fd32c0aa1f0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c13ff10 con 0x7fd32c0a96b0 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042986537 0 0) 0x7fd32c0a9d60 con 0x7fd32c142a20 2024-03-25T18:14:42.919 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.920 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.920 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.920 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:14:42.920 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.918+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c0aa1f0 con 0x7fd32c142a20 2024-03-25T18:14:42.921 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251317096 0 0) 0x7fd318003520 con 0x7fd32c0b4700 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 >> v1:172.21.15.87:6790/0 conn(0x7fd32c0a96b0 legacy=0x7fd32c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 >> v1:172.21.15.87:6789/0 conn(0x7fd32c142a20 legacy=0x7fd32c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 monclient: found mon.noname-c 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0b4ae0 con 0x7fd32c0b4700 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd328001690 con 0x7fd32c0b4700 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd334455640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd334455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.922 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.920+0000 7fd3311ef640 10 monclient: dump: 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.923 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.921+0000 7fd334455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.924 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3280017f0 con 0x7fd32c0b4700 2024-03-25T18:14:42.924 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.924 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 1 -- 172.21.15.171:0/2737646164 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd328001c00 con 0x7fd32c0b4700 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd334455640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd334455640 4 set_mon_vals no callback set 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd3311ef640 10 monclient: dump: 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.925 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd334455640 10 monclient: shutdown 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd334455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.923+0000 7fd334455640 1 -- 172.21.15.171:0/2737646164 >> v1:172.21.15.171:6789/0 conn(0x7fd32c0b4700 legacy=0x7fd32c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 -- 172.21.15.171:0/2737646164 shutdown_connections 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 -- 172.21.15.171:0/2737646164 >> 172.21.15.171:0/2737646164 conn(0x7fd32c0668e0 msgr2=0x7fd32c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 -- 172.21.15.171:0/2737646164 shutdown_connections 2024-03-25T18:14:42.926 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 -- 172.21.15.171:0/2737646164 wait complete. 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 10 monclient: monmap: 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.928 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 librados: starting msgr at 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.924+0000 7fd334455640 1 librados: starting objecter 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 Processor -- start 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 -- start start 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 librados: setting wanted keys 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 librados: calling monclient init 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient: init 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:42.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c0a96b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c0b4700 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1dabd0 con 0x7fd32c0b4700 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1d3fd0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1d4160 con 0x7fd32c142a20 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd334455640 10 monclient(hunting): authenticate will time out at 7787.066387s 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.925+0000 7fd3321f1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd32c0a96b0 0x7fd32c1db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52704/0 (socket says 172.21.15.171:52704) 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd3321f1640 1 -- 172.21.15.171:0/3303690589 learned_addr learned my addr 172.21.15.171:0/3303690589 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter ms_handle_connect 0x7fd32c0a96b0 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter ms_handle_connect 0x7fd32c0b4700 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter ms_handle_connect 0x7fd32c142a20 2024-03-25T18:14:42.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.926+0000 7fd322ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.931 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902716388 0 0) 0x7fd32c1d3fd0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209484851 0 0) 0x7fd32c1dabd0 con 0x7fd32c0b4700 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.928+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c1d3fd0 con 0x7fd32c0b4700 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738784559 0 0) 0x7fd32c1d4160 con 0x7fd32c142a20 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.932 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c1dabd0 con 0x7fd32c142a20 2024-03-25T18:14:42.933 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.929+0000 7fd38a82e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.932+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089374385 0 0) 0x7fd310004eb0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c1d4160 con 0x7fd32c0a96b0 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd328001bc0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.933 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.934 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.933+0000 7fd322ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301137278 0 0) 0x7fd32c1d3fd0 con 0x7fd32c0b4700 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c0b4700 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd31c0017d0 con 0x7fd32c0b4700 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 787740309 0 0) 0x7fd32c1dabd0 con 0x7fd32c142a20 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c1d3fd0 con 0x7fd32c142a20 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd324001cb0 con 0x7fd32c142a20 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.935+0000 7fd322ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:42.937 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.938 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:42.938 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:42.938 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:42.938 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:42.938 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1755519516 0 0) 0x7fd32c1d4160 con 0x7fd32c0a96b0 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 >> v1:172.21.15.87:6790/0 conn(0x7fd32c142a20 legacy=0x7fd32c137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 >> v1:172.21.15.87:6789/0 conn(0x7fd32c0b4700 legacy=0x7fd32c1d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 1 monclient: found mon.b 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c1db770 con 0x7fd32c0a96b0 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:14:42.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 10 monclient: _renew_subs 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c1367c0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 10 monclient: _renew_subs 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c1d4f00 con 0x7fd32c0a96b0 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.936+0000 7fd334455640 1 librados: init done 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.937+0000 7fd334455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.937+0000 7fd334455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.937+0000 7fd334455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:42.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:42.937+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c1d51b0 con 0x7fd32c0a96b0 2024-03-25T18:14:42.977 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:42.977+0000 7fd38a82e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:43.009 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.008+0000 7fdf7be21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:43.054 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.053+0000 7fdf7be21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:43.093 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:43.093+0000 7fd38a82e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3280011f0 con 0x7fd32c0a96b0 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd328004cb0 con 0x7fd32c0a96b0 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: dump: 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.132 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd328005220 con 0x7fd32c0a96b0 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd323fff640 4 set_mon_vals no callback set 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1527689833 0 0) 0x7fd328006910 con 0x7fd32c0a96b0 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 client.4138.objecter ms_dispatch 0x7fd32c000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd328006bf0 con 0x7fd32c0a96b0 2024-03-25T18:14:43.133 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:43.134 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.131+0000 7fd322ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:43.166 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.166+0000 7fdf7be21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:43.193 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:43.192+0000 7fd38a82e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:43.206 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.205+0000 7fd334455640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a6cae3ee-630b-444f-84bd-bd172830fcab", "id": 4}] 2024-03-25T18:14:43.206 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.205+0000 7fd334455640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a6cae3ee-630b-444f-84bd-bd172830fcab", "id": 4}] 2024-03-25T18:14:43.206 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.205+0000 7fd334455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:43.206 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.205+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a6cae3ee-630b-444f-84bd-bd172830fcab", "id": 4} v 0) v1 -- 0x7fd32c04f730 con 0x7fd32c0a96b0 2024-03-25T18:14:43.261 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.260+0000 7fdf7be21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:43.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.339+0000 7fd322ffd640 1 -- 172.21.15.171:0/3303690589 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6cae3ee-630b-444f-84bd-bd172830fcab", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 702565625 0 1697012583) 0x7fd3280064f0 con 0x7fd32c0a96b0 2024-03-25T18:14:43.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.339+0000 7fd322ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a6cae3ee-630b-444f-84bd-bd172830fcab", "id": 4}] 2024-03-25T18:14:43.340 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.339+0000 7fd322ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:43.340 INFO:teuthology.orchestra.run.smithi171.stdout:4 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.340+0000 7fd334455640 10 librados: watch_flush enter 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.340+0000 7fd334455640 10 librados: watch_flush exit 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.340+0000 7fd334455640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.340+0000 7fd334455640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 10 monclient: shutdown 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 >> v1:172.21.15.171:6789/0 conn(0x7fd32c0a96b0 legacy=0x7fd32c1db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.342 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 shutdown_connections 2024-03-25T18:14:43.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 >> 172.21.15.171:0/3303690589 conn(0x7fd32c0668e0 msgr2=0x7fd32c066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:43.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 shutdown_connections 2024-03-25T18:14:43.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 -- 172.21.15.171:0/3303690589 wait complete. 2024-03-25T18:14:43.343 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.341+0000 7fd334455640 1 librados: shutdown 2024-03-25T18:14:43.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 0a7288f1-5ccf-4d22-92fc-69a1d3521ccc 5 2024-03-25T18:14:43.354 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:43.353+0000 7fd38a82e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:43.398 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:43.398+0000 7fd38a82e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:43.420 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.419+0000 7fdf7be21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.450+0000 7f492dc0f640 10 monclient: get_monmap_and_config 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.450+0000 7f492dc0f640 10 monclient: build_initial_monmap 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.450+0000 7f492dc0f640 10 monclient: monmap: 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:43.451453+0000 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:43.451453+0000 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:43.451 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:43.452 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:43.452 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 1 Processor -- start 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 1 -- start start 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient: init 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f492806c2b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:43.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f49280b0100 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4928146300 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492813feb0 con 0x7f49280b0100 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492806d050 con 0x7f492806c2b0 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280b06d0 con 0x7f4928146300 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.451+0000 7f492dc0f640 10 monclient(hunting): authenticate will time out at 7787.592680s 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.452+0000 7f49277fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f492806c2b0 0x7f49280af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52714/0 (socket says 172.21.15.171:52714) 2024-03-25T18:14:43.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.452+0000 7f49277fe640 1 -- 172.21.15.171:0/3272250328 learned_addr learned my addr 172.21.15.171:0/3272250328 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994182848 0 0) 0x7f492806d050 con 0x7f492806c2b0 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910003520 con 0x7f492806c2b0 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015514878 0 0) 0x7f492813feb0 con 0x7f49280b0100 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492806d050 con 0x7f49280b0100 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475249735 0 0) 0x7f49280b06d0 con 0x7f4928146300 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:14:43.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.455+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492813feb0 con 0x7f4928146300 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924722254 0 0) 0x7f4910003520 con 0x7f492806c2b0 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 >> v1:172.21.15.87:6790/0 conn(0x7f4928146300 legacy=0x7f492814db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 >> v1:172.21.15.87:6789/0 conn(0x7f49280b0100 legacy=0x7f4928145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:43.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492806d4a0 con 0x7f492806c2b0 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4918001690 con 0x7f492806c2b0 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f492dc0f640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f492dc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f49267fc640 10 monclient: dump: 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.460 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.458+0000 7f492dc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49180017f0 con 0x7f492806c2b0 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 1 -- 172.21.15.171:0/3272250328 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4918001c00 con 0x7f492806c2b0 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f492dc0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f492dc0f640 4 set_mon_vals no callback set 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.460+0000 7f49267fc640 10 monclient: dump: 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.463 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 10 monclient: shutdown 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 1 -- 172.21.15.171:0/3272250328 >> v1:172.21.15.171:6789/0 conn(0x7f492806c2b0 legacy=0x7f49280af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 1 -- 172.21.15.171:0/3272250328 shutdown_connections 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 1 -- 172.21.15.171:0/3272250328 >> 172.21.15.171:0/3272250328 conn(0x7f49280b5cd0 msgr2=0x7f49280b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 1 -- 172.21.15.171:0/3272250328 shutdown_connections 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 1 -- 172.21.15.171:0/3272250328 wait complete. 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 10 monclient: build_initial_monmap 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.461+0000 7f492dc0f640 10 monclient: monmap: 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 librados: starting msgr at 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 librados: starting objecter 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 Processor -- start 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 -- start start 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 librados: setting wanted keys 2024-03-25T18:14:43.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 librados: calling monclient init 2024-03-25T18:14:43.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient: init 2024-03-25T18:14:43.466 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.463+0000 7fdf7be21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f492806c2b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f49280b0100 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4928146300 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49281407a0 con 0x7f49280b0100 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928140040 con 0x7f4928146300 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492813aab0 con 0x7f492806c2b0 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.462+0000 7f492dc0f640 10 monclient(hunting): authenticate will time out at 7787.603698s 2024-03-25T18:14:43.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f4927fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4928146300 0x7f49281d7330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52718/0 (socket says 172.21.15.171:52718) 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f4927fff640 1 -- 172.21.15.171:0/2414689015 learned_addr learned my addr 172.21.15.171:0/2414689015 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter ms_handle_connect 0x7f4928146300 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter ms_handle_connect 0x7f49280b0100 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter ms_handle_connect 0x7f492806c2b0 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.463+0000 7f490ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036463364 0 0) 0x7f4928140040 con 0x7f4928146300 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914004eb0 con 0x7f4928146300 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628788188 0 0) 0x7f492813aab0 con 0x7f492806c2b0 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4928140040 con 0x7f492806c2b0 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896573971 0 0) 0x7f49281407a0 con 0x7f49280b0100 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:14:43.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.465+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492813aab0 con 0x7f49280b0100 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.469+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598314181 0 0) 0x7f4914004eb0 con 0x7f4928146300 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49281407a0 con 0x7f4928146300 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4920001c70 con 0x7f4928146300 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 949493338 0 0) 0x7f492813aab0 con 0x7f49280b0100 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4914004eb0 con 0x7f49280b0100 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50741336 0 0) 0x7f4928140040 con 0x7f492806c2b0 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492813aab0 con 0x7f492806c2b0 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4918001b00 con 0x7f492806c2b0 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.470+0000 7f490ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.473 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.471+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f491c001950 con 0x7f49280b0100 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.471+0000 7f490ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.471+0000 7f490ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.471+0000 7f490ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.474 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.475 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.476 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.472+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902396163 0 0) 0x7f492813aab0 con 0x7f492806c2b0 2024-03-25T18:14:43.476 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 >> v1:172.21.15.171:6789/0 conn(0x7f4928146300 legacy=0x7f49281d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.476 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 >> v1:172.21.15.87:6789/0 conn(0x7f49280b0100 legacy=0x7f4928139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 1 monclient: found mon.c 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492813ac40 con 0x7f492806c2b0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f490ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 5 monclient: authenticate success, global_id 4145 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _renew_subs 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492813a5d0 con 0x7f492806c2b0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _renew_subs 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:14:43.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49281d7c20 con 0x7f492806c2b0 2024-03-25T18:14:43.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 1 librados: init done 2024-03-25T18:14:43.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:43.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:43.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:14:43.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.473+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49280682a0 con 0x7f492806c2b0 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918001ca0 con 0x7f492806c2b0 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4918004a90 con 0x7f492806c2b0 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 10 monclient: dump: 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:43.523 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4918004f80 con 0x7f492806c2b0 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2623881432 0 0) 0x7f4918006660 con 0x7f492806c2b0 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 10 client.4145.objecter ms_dispatch 0x7f4928000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 3 client.4145.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 3 client.4145.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f4925ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:43.524 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.522+0000 7f490ffff640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T18:14:43.559 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:43.558+0000 7fd38a82e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:43.624 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:43.624+0000 7fdf7be21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:43.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.687+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4918006190 con 0x7f492806c2b0 2024-03-25T18:14:43.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.687+0000 7f490ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:43.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.687+0000 7f490ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:43.839 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.839+0000 7f492dc0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0a7288f1-5ccf-4d22-92fc-69a1d3521ccc", "id": 5}] 2024-03-25T18:14:43.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.839+0000 7f492dc0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0a7288f1-5ccf-4d22-92fc-69a1d3521ccc", "id": 5}] 2024-03-25T18:14:43.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.839+0000 7f492dc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:14:43.840 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.839+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd new", "uuid": "0a7288f1-5ccf-4d22-92fc-69a1d3521ccc", "id": 5} v 0) v1 -- 0x7f4928141a20 con 0x7f492806c2b0 2024-03-25T18:14:43.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.944+0000 7f490ffff640 1 -- 172.21.15.171:0/2414689015 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a7288f1-5ccf-4d22-92fc-69a1d3521ccc", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1936921689 0 1988412176) 0x7f4918006400 con 0x7f492806c2b0 2024-03-25T18:14:43.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.944+0000 7f490ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0a7288f1-5ccf-4d22-92fc-69a1d3521ccc", "id": 5}] 2024-03-25T18:14:43.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.944+0000 7f490ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:43.945 INFO:teuthology.orchestra.run.smithi171.stdout:5 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 10 librados: watch_flush enter 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 10 librados: watch_flush exit 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 10 client.4145.objecter successfully canceled tick 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 10 monclient: shutdown 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:43.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 >> v1:172.21.15.87:6790/0 conn(0x7f492806c2b0 legacy=0x7f4928141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 shutdown_connections 2024-03-25T18:14:43.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 >> 172.21.15.171:0/2414689015 conn(0x7f49280b5cd0 msgr2=0x7f49280aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:43.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 shutdown_connections 2024-03-25T18:14:43.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 -- 172.21.15.171:0/2414689015 wait complete. 2024-03-25T18:14:43.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:43.946+0000 7f492dc0f640 1 librados: shutdown 2024-03-25T18:14:43.960 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 89fa7866-1bf5-408f-924d-0d7fba1d5565 6 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.061+0000 7ff66123e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.061+0000 7ff66123e640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.061+0000 7ff66123e640 10 monclient: monmap: 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:44.062658+0000 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:44.062658+0000 2024-03-25T18:14:44.062 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.063 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.063 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.063 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:44.063 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:44.063 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 1 Processor -- start 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 1 -- start start 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient: init 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-a con 0x7ff65c0b4700 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-b con 0x7ff65c142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-c con 0x7ff65c0a96b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0a9d60 con 0x7ff65c0b4700 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c13ff10 con 0x7ff65c0a96b0 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0aa1f0 con 0x7ff65c142a20 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.062+0000 7ff66123e640 10 monclient(hunting): authenticate will time out at 7788.203700s 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.063+0000 7ff65b59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff65c0a96b0 0x7ff65c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52724/0 (socket says 172.21.15.171:52724) 2024-03-25T18:14:44.065 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.063+0000 7ff65b59e640 1 -- 172.21.15.171:0/1893130390 learned_addr learned my addr 172.21.15.171:0/1893130390 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228212080 0 0) 0x7ff65c13ff10 con 0x7ff65c0a96b0 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:14:44.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.065+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff648003520 con 0x7ff65c0a96b0 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402143497 0 0) 0x7ff65c0a9d60 con 0x7ff65c0b4700 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:14:44.067 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c13ff10 con 0x7ff65c0b4700 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956422396 0 0) 0x7ff65c0aa1f0 con 0x7ff65c142a20 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:14:44.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.066+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c0a9d60 con 0x7ff65c142a20 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216549706 0 0) 0x7ff648003520 con 0x7ff65c0a96b0 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 >> v1:172.21.15.87:6790/0 conn(0x7ff65c142a20 legacy=0x7ff65c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 >> v1:172.21.15.87:6789/0 conn(0x7ff65c0b4700 legacy=0x7ff65c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 monclient: found mon.noname-c 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.070 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0b4ae0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff654001a10 con 0x7ff65c0a96b0 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff66123e640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff66123e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff65a59c640 10 monclient: dump: 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.071 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.072 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.072 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.072 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.072 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.072 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.069+0000 7ff66123e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff654001b70 con 0x7ff65c0a96b0 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 1 -- 172.21.15.171:0/1893130390 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff654001f80 con 0x7ff65c0a96b0 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff66123e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff66123e640 4 set_mon_vals no callback set 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.070+0000 7ff65a59c640 10 monclient: dump: 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.073 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 10 monclient: shutdown 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 1 -- 172.21.15.171:0/1893130390 >> v1:172.21.15.171:6789/0 conn(0x7ff65c0a96b0 legacy=0x7ff65c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 1 -- 172.21.15.171:0/1893130390 shutdown_connections 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 1 -- 172.21.15.171:0/1893130390 >> 172.21.15.171:0/1893130390 conn(0x7ff65c0668e0 msgr2=0x7ff65c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 1 -- 172.21.15.171:0/1893130390 shutdown_connections 2024-03-25T18:14:44.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.071+0000 7ff66123e640 1 -- 172.21.15.171:0/1893130390 wait complete. 2024-03-25T18:14:44.076 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.076 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 10 monclient: monmap: 2024-03-25T18:14:44.076 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:44.076 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 librados: starting msgr at 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 librados: starting objecter 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 Processor -- start 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 -- start start 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 librados: setting wanted keys 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 1 librados: calling monclient init 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.073+0000 7ff66123e640 10 monclient: init 2024-03-25T18:14:44.077 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-b con 0x7ff65c0a96b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-a con 0x7ff65c0b4700 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient(hunting): picked mon.noname-c con 0x7ff65c142a20 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c13b3e0 con 0x7ff65c0b4700 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0ae520 con 0x7ff65c0a96b0 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0ab880 con 0x7ff65c142a20 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.074+0000 7ff66123e640 10 monclient(hunting): authenticate will time out at 7788.215428s 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff65ad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff65c0a96b0 0x7ff65c0ac810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52732/0 (socket says 172.21.15.171:52732) 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff65ad9d640 1 -- 172.21.15.171:0/3778555411 learned_addr learned my addr 172.21.15.171:0/3778555411 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter ms_handle_connect 0x7ff65c0a96b0 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter ms_handle_connect 0x7ff65c142a20 2024-03-25T18:14:44.078 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter ms_handle_connect 0x7ff65c0b4700 2024-03-25T18:14:44.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.075+0000 7ff653fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267502540 0 0) 0x7ff65c0ae520 con 0x7ff65c0a96b0 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c004eb0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997929343 0 0) 0x7ff65c0ab880 con 0x7ff65c142a20 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.077+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c0ae520 con 0x7ff65c142a20 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661569709 0 0) 0x7ff65c13b3e0 con 0x7ff65c0b4700 2024-03-25T18:14:44.080 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.081 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.081 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.081 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:14:44.081 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.078+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c0ab880 con 0x7ff65c0b4700 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3854655459 0 0) 0x7ff63c004eb0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff65c13b3e0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff64c001a30 con 0x7ff65c0a96b0 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 10 monclient(hunting): dump: 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.083 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.084 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.084 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.084 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.084 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.084 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.082+0000 7ff653fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.083+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3268535871 0 0) 0x7ff65c0ab880 con 0x7ff65c0b4700 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.083+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff63c004eb0 con 0x7ff65c0b4700 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034431021 0 0) 0x7ff65c0ae520 con 0x7ff65c142a20 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff65c0ab880 con 0x7ff65c142a20 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff654001e80 con 0x7ff65c142a20 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): dump: 2024-03-25T18:14:44.085 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff6440018f0 con 0x7ff65c0b4700 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.084+0000 7ff653fff640 10 monclient(hunting): dump: 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.087 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.088 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4179352405 0 0) 0x7ff65c13b3e0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.088 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 >> v1:172.21.15.87:6790/0 conn(0x7ff65c142a20 legacy=0x7ff65c0b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.088 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 >> v1:172.21.15.87:6789/0 conn(0x7ff65c0b4700 legacy=0x7ff65c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.088 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.088 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 1 monclient: found mon.b 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0aba10 con 0x7ff65c0a96b0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _renew_subs 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c13a490 con 0x7ff65c0a96b0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _renew_subs 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c0ad430 con 0x7ff65c0a96b0 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 1 librados: init done 2024-03-25T18:14:44.089 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.090 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.090 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.090 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.086+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c04f730 con 0x7ff65c0a96b0 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff64c001c70 con 0x7ff65c0a96b0 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff64c003a60 con 0x7ff65c0a96b0 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: dump: 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff64c003fd0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff659d9b640 4 set_mon_vals no callback set 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 577574736 0 0) 0x7ff64c006810 con 0x7ff65c0a96b0 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 client.4150.objecter ms_dispatch 0x7ff65c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:14:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff64c018dc0 con 0x7ff65c0a96b0 2024-03-25T18:14:44.193 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.193 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.190+0000 7ff653fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:44.264 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.263+0000 7ff66123e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89fa7866-1bf5-408f-924d-0d7fba1d5565", "id": 6}] 2024-03-25T18:14:44.264 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.263+0000 7ff66123e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89fa7866-1bf5-408f-924d-0d7fba1d5565", "id": 6}] 2024-03-25T18:14:44.264 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.263+0000 7ff66123e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.264 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.263+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89fa7866-1bf5-408f-924d-0d7fba1d5565", "id": 6} v 0) v1 -- 0x7ff65c142220 con 0x7ff65c0a96b0 2024-03-25T18:14:44.321 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff653fff640 1 -- 172.21.15.171:0/3778555411 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89fa7866-1bf5-408f-924d-0d7fba1d5565", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3645920741 0 1113811849) 0x7ff64c006220 con 0x7ff65c0a96b0 2024-03-25T18:14:44.321 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff653fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89fa7866-1bf5-408f-924d-0d7fba1d5565", "id": 6}] 2024-03-25T18:14:44.321 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff653fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:44.322 INFO:teuthology.orchestra.run.smithi171.stdout:6 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff66123e640 10 librados: watch_flush enter 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff66123e640 10 librados: watch_flush exit 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff66123e640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.321+0000 7ff66123e640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 10 monclient: shutdown 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 >> v1:172.21.15.171:6789/0 conn(0x7ff65c0a96b0 legacy=0x7ff65c0ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 shutdown_connections 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 >> 172.21.15.171:0/3778555411 conn(0x7ff65c0668e0 msgr2=0x7ff65c13c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 shutdown_connections 2024-03-25T18:14:44.324 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 -- 172.21.15.171:0/3778555411 wait complete. 2024-03-25T18:14:44.324 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.322+0000 7ff66123e640 1 librados: shutdown 2024-03-25T18:14:44.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 526fd20b-e3d4-43e6-b854-52268cb06e06 7 2024-03-25T18:14:44.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.433+0000 7f8e1284a640 10 monclient: get_monmap_and_config 2024-03-25T18:14:44.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.433+0000 7f8e1284a640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.433+0000 7f8e1284a640 10 monclient: monmap: 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:14:44.434241+0000 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:14:44.434241+0000 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:44.434 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.433+0000 7f8e1284a640 1 Processor -- start 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 1 -- start start 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient: init 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0c0a9920 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e0c0b45a0 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e0c146450 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c06b100 con 0x7f8e0c0a9920 2024-03-25T18:14:44.436 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0b2940 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c13fec0 con 0x7f8e0c146450 2024-03-25T18:14:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.434+0000 7f8e1284a640 10 monclient(hunting): authenticate will time out at 7788.575619s 2024-03-25T18:14:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.435+0000 7f8e0b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8e0c0b45a0 0x7f8e0c145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52744/0 (socket says 172.21.15.171:52744) 2024-03-25T18:14:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.435+0000 7f8e0b7fe640 1 -- 172.21.15.171:0/466960754 learned_addr learned my addr 172.21.15.171:0/466960754 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:44.438 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599585019 0 0) 0x7f8e0c0b2940 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.438 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.438 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.438 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.438 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df4003520 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289510741 0 0) 0x7f8e0c13fec0 con 0x7f8e0c146450 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.437+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c0b2940 con 0x7f8e0c146450 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523438434 0 0) 0x7f8e0c06b100 con 0x7f8e0c0a9920 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:14:44.439 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.438+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c13fec0 con 0x7f8e0c0a9920 2024-03-25T18:14:44.441 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145840644 0 0) 0x7f8df4003520 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.441 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 >> v1:172.21.15.87:6790/0 conn(0x7f8e0c146450 legacy=0x7f8e0c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.441 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 >> v1:172.21.15.87:6789/0 conn(0x7f8e0c0a9920 legacy=0x7f8e0c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.441 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 monclient: found mon.noname-c 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c140050 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e00001690 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e0affd640 10 monclient: dump: 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.442 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e1284a640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:14:44.443 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.440+0000 7f8e1284a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:44.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e000017f0 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 1 -- 172.21.15.171:0/466960754 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e00001a60 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.446 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e1284a640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e1284a640 4 set_mon_vals no callback set 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.442+0000 7f8e0affd640 10 monclient: dump: 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 10 monclient: shutdown 2024-03-25T18:14:44.447 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 1 -- 172.21.15.171:0/466960754 >> v1:172.21.15.171:6789/0 conn(0x7f8e0c0b45a0 legacy=0x7f8e0c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 1 -- 172.21.15.171:0/466960754 shutdown_connections 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 1 -- 172.21.15.171:0/466960754 >> 172.21.15.171:0/466960754 conn(0x7f8e0c0b7f40 msgr2=0x7f8e0c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 1 -- 172.21.15.171:0/466960754 shutdown_connections 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 1 -- 172.21.15.171:0/466960754 wait complete. 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.443+0000 7f8e1284a640 10 monclient: monmap: 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-03-25T18:14:44.448 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 librados: starting msgr at 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 librados: starting objecter 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 Processor -- start 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 -- start start 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 librados: setting wanted keys 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 1 librados: calling monclient init 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 10 monclient: init 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.444+0000 7f8e1284a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0c0a9920 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e0c0b45a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.449 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e0c146450 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0704c0 con 0x7f8e0c0a9920 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c06f400 con 0x7f8e0c146450 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0709e0 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e1284a640 10 monclient(hunting): authenticate will time out at 7788.585849s 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e10de7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8e0c146450 0x7f8e0c1daee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52748/0 (socket says 172.21.15.171:52748) 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e10de7640 1 -- 172.21.15.171:0/3914753332 learned_addr learned my addr 172.21.15.171:0/3914753332 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e08ff9640 10 client.?.objecter ms_handle_connect 0x7f8e0c146450 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.445+0000 7f8e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.446+0000 7f8e08ff9640 10 client.?.objecter ms_handle_connect 0x7f8e0c0a9920 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.446+0000 7f8e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.446+0000 7f8e08ff9640 10 client.?.objecter ms_handle_connect 0x7f8e0c0b45a0 2024-03-25T18:14:44.450 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.446+0000 7f8e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736081059 0 0) 0x7f8e0c06f400 con 0x7f8e0c146450 2024-03-25T18:14:44.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.447+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df8004eb0 con 0x7f8e0c146450 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147228049 0 0) 0x7f8e0c0709e0 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c06f400 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858612156 0 0) 0x7f8e0c0704c0 con 0x7f8e0c0a9920 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:14:44.453 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.448+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c0709e0 con 0x7f8e0c0a9920 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915303654 0 0) 0x7f8df8004eb0 con 0x7f8e0c146450 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e0c0704c0 con 0x7f8e0c146450 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e04001c90 con 0x7f8e0c146450 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.454 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1125279136 0 0) 0x7f8e0c06f400 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.452+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8df8004eb0 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e00001960 con 0x7f8e0c0b45a0 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.455 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4271543363 0 0) 0x7f8e0c0709e0 con 0x7f8e0c0a9920 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e0c06f400 con 0x7f8e0c0a9920 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8dfc001830 con 0x7f8e0c0a9920 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.453+0000 7f8e08ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.456 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.457 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.457 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.457 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.457 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.457 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.458 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312773190 0 0) 0x7f8e0c0704c0 con 0x7f8e0c146450 2024-03-25T18:14:44.458 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 >> v1:172.21.15.87:6790/0 conn(0x7f8e0c0b45a0 legacy=0x7f8e0c1da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.458 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 >> v1:172.21.15.87:6789/0 conn(0x7f8e0c0a9920 legacy=0x7f8e0c1ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 1 monclient: found mon.b 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c071ef0 con 0x7f8e0c146450 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e1284a640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e1284a640 10 monclient: _renew_subs 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e1284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.455+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c06ff10 con 0x7f8e0c146450 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 monclient: _renew_subs 2024-03-25T18:14:44.459 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c13c080 con 0x7f8e0c146450 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 1 librados: init done 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.456+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c072140 con 0x7f8e0c146450 2024-03-25T18:14:44.460 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:44.459+0000 7fd38a82e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e04003970 con 0x7f8e0c146450 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e04003d80 con 0x7f8e0c146450 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: dump: 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.557 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 3 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8e04004330 con 0x7f8e0c146450 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e0a7fc640 4 set_mon_vals no callback set 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3660452008 0 0) 0x7f8e04006ba0 con 0x7f8e0c146450 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 client.4156.objecter ms_dispatch 0x7f8e0c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8e04018e00 con 0x7f8e0c146450 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.556+0000 7f8e08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:44.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.628+0000 7f8e1284a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "526fd20b-e3d4-43e6-b854-52268cb06e06", "id": 7}] 2024-03-25T18:14:44.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.628+0000 7f8e1284a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "526fd20b-e3d4-43e6-b854-52268cb06e06", "id": 7}] 2024-03-25T18:14:44.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.628+0000 7f8e1284a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:44.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.628+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "526fd20b-e3d4-43e6-b854-52268cb06e06", "id": 7} v 0) v1 -- 0x7f8e0c04f730 con 0x7f8e0c146450 2024-03-25T18:14:44.709 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.708+0000 7f8e08ff9640 1 -- 172.21.15.171:0/3914753332 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "526fd20b-e3d4-43e6-b854-52268cb06e06", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 579514128 0 1371665406) 0x7f8e040045e0 con 0x7f8e0c146450 2024-03-25T18:14:44.709 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.708+0000 7f8e08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "526fd20b-e3d4-43e6-b854-52268cb06e06", "id": 7}] 2024-03-25T18:14:44.709 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.708+0000 7f8e08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:44.709 INFO:teuthology.orchestra.run.smithi171.stdout:7 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 10 librados: watch_flush enter 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 10 librados: watch_flush exit 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 10 monclient: shutdown 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 >> v1:172.21.15.171:6789/0 conn(0x7f8e0c146450 legacy=0x7f8e0c1daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.710 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 shutdown_connections 2024-03-25T18:14:44.711 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 >> 172.21.15.171:0/3914753332 conn(0x7f8e0c0b7f40 msgr2=0x7f8e0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:44.711 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 shutdown_connections 2024-03-25T18:14:44.711 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 -- 172.21.15.171:0/3914753332 wait complete. 2024-03-25T18:14:44.711 INFO:teuthology.orchestra.run.smithi171.stderr:2024-03-25T18:14:44.709+0000 7f8e1284a640 1 librados: shutdown 2024-03-25T18:14:44.719 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:44.718+0000 7fdf7be21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:44.720 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.720 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:14:44.721 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:14:44.723 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:14:44.723 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.723 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:14:44.723 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:14:44.725 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:14:44.725 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.725 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:14:44.725 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:14:44.728 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:14:44.728 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.728 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:14:44.728 DEBUG:teuthology.orchestra.run.smithi087:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:14:44.730 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:14:44.731 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.731 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:14:44.731 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:14:44.733 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:14:44.733 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.733 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:14:44.733 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:14:44.735 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:14:44.735 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.735 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:14:44.736 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:14:44.738 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:14:44.738 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:44.738 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:14:44.738 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:14:44.741 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:14:44.741 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:44.757 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:44.756+0000 7fd38a82e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:44.821 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:44.820+0000 7fd38a82e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.882+0000 7fd2f350e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.882+0000 7fd2f350e640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.882+0000 7fd2f350e640 10 monclient: monmap: 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:14:44.883221+0000 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:14:44.883221+0000 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:44.884 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 1 Processor -- start 2024-03-25T18:14:44.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 1 -- start start 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 10 monclient: init 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ec0b4700 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.883+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ec142a20 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ec0a96b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0a9d60 con 0x7fd2ec142a20 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec13ff10 con 0x7fd2ec0b4700 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0aa1f0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f350e640 10 monclient(hunting): authenticate will time out at 7800.179109s 2024-03-25T18:14:44.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f12aa640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd2ec0b4700 0x7fd2ec142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:59878/0 (socket says 172.21.15.87:59878) 2024-03-25T18:14:44.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.884+0000 7fd2f12aa640 1 -- 172.21.15.87:0/441616878 learned_addr learned my addr 172.21.15.87:0/441616878 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:44.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182563436 0 0) 0x7fd2ec0a9d60 con 0x7fd2ec142a20 2024-03-25T18:14:44.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:14:44.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.887+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d8003520 con 0x7fd2ec142a20 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360644843 0 0) 0x7fd2ec0aa1f0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec0a9d60 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.888+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812048248 0 0) 0x7fd2ec13ff10 con 0x7fd2ec0b4700 2024-03-25T18:14:44.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.889+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.889+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.889+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.889+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:14:44.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.889+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec0aa1f0 con 0x7fd2ec0b4700 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.890+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 913036029 0 0) 0x7fd2d8003520 con 0x7fd2ec142a20 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.890+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 >> v1:172.21.15.87:6790/0 conn(0x7fd2ec0a96b0 legacy=0x7fd2ec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 >> v1:172.21.15.171:6789/0 conn(0x7fd2ec0b4700 legacy=0x7fd2ec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 1 monclient: found mon.noname-a 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0b4ae0 con 0x7fd2ec142a20 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd2dc001650 con 0x7fd2ec142a20 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2e3fff640 10 monclient: dump: 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:44.897 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2f350e640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.891+0000 7fd2f350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.892+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2dc0018a0 con 0x7fd2ec142a20 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2e3fff640 1 -- 172.21.15.87:0/441616878 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd2dc001c30 con 0x7fd2ec142a20 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2e3fff640 10 monclient: dump: 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.898 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 4 set_mon_vals no callback set 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 10 monclient: shutdown 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 1 -- 172.21.15.87:0/441616878 >> v1:172.21.15.87:6789/0 conn(0x7fd2ec142a20 legacy=0x7fd2ec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 1 -- 172.21.15.87:0/441616878 shutdown_connections 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.893+0000 7fd2f350e640 1 -- 172.21.15.87:0/441616878 >> 172.21.15.87:0/441616878 conn(0x7fd2ec0668e0 msgr2=0x7fd2ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.894+0000 7fd2f350e640 1 -- 172.21.15.87:0/441616878 shutdown_connections 2024-03-25T18:14:44.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.894+0000 7fd2f350e640 1 -- 172.21.15.87:0/441616878 wait complete. 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.894+0000 7fd2f350e640 10 monclient: build_initial_monmap 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.894+0000 7fd2f350e640 10 monclient: monmap: 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 0 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 0.000000 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:created 0.000000 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 1 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 librados: starting msgr at 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 librados: starting objecter 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 Processor -- start 2024-03-25T18:14:44.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 -- start start 2024-03-25T18:14:44.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 librados: setting wanted keys 2024-03-25T18:14:44.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 1 librados: calling monclient init 2024-03-25T18:14:44.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.895+0000 7fd2f350e640 10 monclient: init 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ec0a96b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ec0b4700 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ec142a20 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0b0e60 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec14ec70 con 0x7fd2ec142a20 2024-03-25T18:14:44.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec14f810 con 0x7fd2ec0b4700 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.896+0000 7fd2f350e640 10 monclient(hunting): authenticate will time out at 7800.191225s 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.897+0000 7fd2f12aa640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fd2ec0a96b0 0x7fd2ec0b1560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:37954/0 (socket says 172.21.15.87:37954) 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.897+0000 7fd2f12aa640 1 -- 172.21.15.87:0/3982127788 learned_addr learned my addr 172.21.15.87:0/3982127788 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.897+0000 7fd2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2ec0a96b0 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.897+0000 7fd2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.898+0000 7fd2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2ec142a20 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.898+0000 7fd2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.898+0000 7fd2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2ec0b4700 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.898+0000 7fd2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112750815 0 0) 0x7fd2ec0b0e60 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:14:44.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.899+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d0004eb0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207050779 0 0) 0x7fd2ec14ec70 con 0x7fd2ec142a20 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec0b0e60 con 0x7fd2ec142a20 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946335628 0 0) 0x7fd2ec14f810 con 0x7fd2ec0b4700 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:14:44.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.901+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec14ec70 con 0x7fd2ec0b4700 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4257992131 0 0) 0x7fd2d0004eb0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ec14f810 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd2e80018a0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.905 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.904+0000 7fd2e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1911672438 0 0) 0x7fd2ec14f810 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 >> v1:172.21.15.87:6790/0 conn(0x7fd2ec0b4700 legacy=0x7fd2ec14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 >> v1:172.21.15.171:6789/0 conn(0x7fd2ec142a20 legacy=0x7fd2ec12ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 1 monclient: found mon.a 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec14e8b0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:44.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2f350e640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2f350e640 10 monclient: _renew_subs 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2f350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.908+0000 7fd2f350e640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec13c150 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.909+0000 7fd2f350e640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.909+0000 7fd2f350e640 10 monclient: _renew_subs 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.909+0000 7fd2f350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.909+0000 7fd2f350e640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec0ae8e0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.909+0000 7fd2f350e640 1 librados: init done 2024-03-25T18:14:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.910+0000 7fd2d77fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.910+0000 7fd2d77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.910+0000 7fd2d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.910+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec13c4f0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.911+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2e80047e0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.911+0000 7fd2e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:44.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.911+0000 7fd2e2ffd640 4 set_mon_vals no callback set 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.952+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd2e80018a0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.952+0000 7fd2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.952+0000 7fd2e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.952+0000 7fd2e1ffb640 10 monclient: dump: 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:epoch 1 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:44.953 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:14:44.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.952+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd2e8004e60 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.966+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd2e80058d0 con 0x7fd2ec0a96b0 2024-03-25T18:14:44.967 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.966+0000 7fd2e1ffb640 10 client.4167.objecter ms_dispatch 0x7fd2ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:44.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.966+0000 7fd2e1ffb640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:44.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.966+0000 7fd2e1ffb640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:44.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:44.966+0000 7fd2e1ffb640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T18:14:45.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.011+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd2e8005ae0 con 0x7fd2ec0a96b0 2024-03-25T18:14:45.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.011+0000 7fd2e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.011+0000 7fd2e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:45.029 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:45.028+0000 7fd38a82e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:45.067 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:45.066+0000 7fdf7be21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:45.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.098+0000 7fd2d77fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.098+0000 7fd2d77fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.098+0000 7fd2d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:45.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.098+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2ec04f730 con 0x7fd2ec0a96b0 2024-03-25T18:14:45.150 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:45.149+0000 7fdf7be21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:45.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.238+0000 7fd2e1ffb640 1 -- 172.21.15.87:0/3982127788 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd2e80051c0 con 0x7fd2ec0a96b0 2024-03-25T18:14:45.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.238+0000 7fd2e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:45.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.238+0000 7fd2e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:45.246 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-25T18:14:45.246 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:45.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 10 librados: watch_flush enter 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 10 librados: watch_flush exit 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 10 client.4167.objecter successfully canceled tick 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 10 monclient: shutdown 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 >> v1:172.21.15.87:6789/0 conn(0x7fd2ec0a96b0 legacy=0x7fd2ec0b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 shutdown_connections 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 >> 172.21.15.87:0/3982127788 conn(0x7fd2ec0668e0 msgr2=0x7fd2ec0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.247+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 shutdown_connections 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.248+0000 7fd2d77fe640 1 -- 172.21.15.87:0/3982127788 wait complete. 2024-03-25T18:14:45.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-03-25T18:14:45.248+0000 7fd2d77fe640 1 librados: shutdown 2024-03-25T18:14:45.265 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:14:45.265 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:14:45.290 INFO:tasks.ceph.mgr.y.smithi087.stderr:2024-03-25T18:14:45.290+0000 7fdf7be21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:46.071 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:46.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.204+0000 7f1167606640 10 monclient: get_monmap_and_config 2024-03-25T18:14:46.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.204+0000 7f1167606640 10 monclient: build_initial_monmap 2024-03-25T18:14:46.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.204+0000 7f1167606640 10 monclient: monmap: 2024-03-25T18:14:46.205 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:14:46.205 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:14:46.205626+0000 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:14:46.205626+0000 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:46.206 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.205+0000 7f1167606640 1 Processor -- start 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.205+0000 7f1167606640 1 -- start start 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.205+0000 7f1167606640 10 monclient: init 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient(hunting): picked mon.noname-a con 0x7f11600a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient(hunting): picked mon.noname-c con 0x7f116013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient(hunting): picked mon.noname-b con 0x7f11601430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1160142520 con 0x7f11600a98f0 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11600b28a0 con 0x7f116013f710 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11600b2a30 con 0x7f11601430c0 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.206+0000 7f1167606640 10 monclient(hunting): authenticate will time out at 7801.501681s 2024-03-25T18:14:46.209 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.207+0000 7f1164ba1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f116013f710 0x7f11601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:59888/0 (socket says 172.21.15.87:59888) 2024-03-25T18:14:46.209 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.207+0000 7f1164ba1640 1 -- 172.21.15.87:0/3071995096 learned_addr learned my addr 172.21.15.87:0/3071995096 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:46.211 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347342094 0 0) 0x7f11600b28a0 con 0x7f116013f710 2024-03-25T18:14:46.211 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:46.211 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:46.211 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:46.212 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:14:46.212 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.210+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f114c003520 con 0x7f116013f710 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255993106 0 0) 0x7f114c003520 con 0x7f116013f710 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 >> v1:172.21.15.87:6790/0 conn(0x7f11601430c0 legacy=0x7f116014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 >> v1:172.21.15.87:6789/0 conn(0x7f11600a98f0 legacy=0x7f11600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 1 monclient: found mon.noname-c 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11600b44c0 con 0x7f116013f710 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1157fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1167606640 5 monclient: authenticate success, global_id 4177 2024-03-25T18:14:46.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.216+0000 7f1167606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:46.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.217+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1150001690 con 0x7f116013f710 2024-03-25T18:14:46.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.217+0000 7f1157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:46.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.217+0000 7f1157fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.217+0000 7f1157fff640 10 monclient: dump: 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.218+0000 7f1167606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1150001960 con 0x7f116013f710 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1167606640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 1 -- 172.21.15.87:0/3071995096 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1150001cf0 con 0x7f116013f710 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1167606640 4 set_mon_vals no callback set 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.220+0000 7f1157fff640 10 monclient: dump: 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.223 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 10 monclient: shutdown 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 1 -- 172.21.15.87:0/3071995096 >> v1:172.21.15.171:6789/0 conn(0x7f116013f710 legacy=0x7f11601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 1 -- 172.21.15.87:0/3071995096 shutdown_connections 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 1 -- 172.21.15.87:0/3071995096 >> 172.21.15.87:0/3071995096 conn(0x7f11600b7ea0 msgr2=0x7f11600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 1 -- 172.21.15.87:0/3071995096 shutdown_connections 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.221+0000 7f1167606640 1 -- 172.21.15.87:0/3071995096 wait complete. 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 10 monclient: build_initial_monmap 2024-03-25T18:14:46.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 10 monclient: monmap: 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 librados: starting msgr at 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 librados: starting objecter 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 Processor -- start 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 -- start start 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 librados: setting wanted keys 2024-03-25T18:14:46.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 1 librados: calling monclient init 2024-03-25T18:14:46.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.222+0000 7f1167606640 10 monclient: init 2024-03-25T18:14:46.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:46.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:46.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient(hunting): picked mon.noname-a con 0x7f11600a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:46.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient(hunting): picked mon.noname-b con 0x7f116013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:46.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient(hunting): picked mon.noname-c con 0x7f11601430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11601d77a0 con 0x7f11600a98f0 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11601d8340 con 0x7f116013f710 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11601397c0 con 0x7f11601430c0 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.223+0000 7f1167606640 10 monclient(hunting): authenticate will time out at 7801.518469s 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.226+0000 7f11653a2640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f11600a98f0 0x7f1160138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:37974/0 (socket says 172.21.15.87:37974) 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.226+0000 7f11653a2640 1 -- 172.21.15.87:0/810704453 learned_addr learned my addr 172.21.15.87:0/810704453 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.227+0000 7f1155ffb640 10 client.?.objecter ms_handle_connect 0x7f11601430c0 2024-03-25T18:14:46.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.227+0000 7f1155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:46.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.228+0000 7f1155ffb640 10 client.?.objecter ms_handle_connect 0x7f116013f710 2024-03-25T18:14:46.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.228+0000 7f1155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:46.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.229+0000 7f1155ffb640 10 client.?.objecter ms_handle_connect 0x7f11600a98f0 2024-03-25T18:14:46.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.229+0000 7f1155ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617421869 0 0) 0x7f11601d77a0 con 0x7f11600a98f0 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:14:46.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.233+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1144004eb0 con 0x7f11600a98f0 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2181070806 0 0) 0x7f1144004eb0 con 0x7f11600a98f0 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11601d77a0 con 0x7f11600a98f0 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f115c001960 con 0x7f11600a98f0 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:46.245 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:46.246 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:46.246 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:46.246 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:46.246 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.246 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.244+0000 7f1155ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480565747 0 0) 0x7f11601d77a0 con 0x7f11600a98f0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 >> v1:172.21.15.87:6790/0 conn(0x7f11601430c0 legacy=0x7f1160133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 >> v1:172.21.15.171:6789/0 conn(0x7f116013f710 legacy=0x7f11601d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 1 monclient: found mon.a 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160137a00 con 0x7f11600a98f0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:46.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 10 monclient: _renew_subs 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1160137c50 con 0x7f11600a98f0 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 10 monclient: _renew_subs 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1160134800 con 0x7f11600a98f0 2024-03-25T18:14:46.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.312+0000 7f1167606640 1 librados: init done 2024-03-25T18:14:46.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.313+0000 7f1167606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:46.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.313+0000 7f1167606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:46.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.313+0000 7f1167606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:46.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.313+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116014f2d0 con 0x7f11600a98f0 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f115c0035f0 con 0x7f11600a98f0 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f115c003980 con 0x7f11600a98f0 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f1155ffb640 10 monclient: dump: 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:46.463 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.462+0000 7f11577fe640 4 set_mon_vals no callback set 2024-03-25T18:14:46.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.463+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 946068345 0 0) 0x7f115c003ba0 con 0x7f11600a98f0 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f115c015690 con 0x7f11600a98f0 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 10 client.4173.objecter ms_dispatch 0x7f1160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f115c014950 con 0x7f11600a98f0 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.474+0000 7f1155ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:46.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.590+0000 7f1167606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.590+0000 7f1167606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.590+0000 7f1167606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:14:46.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.590+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11600627b0 con 0x7f11600a98f0 2024-03-25T18:14:46.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.681+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 4172344926 0 0) 0x7f115c044d50 con 0x7f11600a98f0 2024-03-25T18:14:46.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.759+0000 7f1155ffb640 1 -- 172.21.15.87:0/810704453 <== mon.0 v1:172.21.15.87:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f115c00d340 con 0x7f11600a98f0 2024-03-25T18:14:46.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.759+0000 7f1155ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.759+0000 7f1155ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:46.761 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:14:46.761 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 10 librados: watch_flush enter 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 10 librados: watch_flush exit 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 10 monclient: shutdown 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:46.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 >> v1:172.21.15.87:6789/0 conn(0x7f11600a98f0 legacy=0x7f1160138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 shutdown_connections 2024-03-25T18:14:46.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 >> 172.21.15.87:0/810704453 conn(0x7f11600b7ea0 msgr2=0x7f1160074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:46.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 shutdown_connections 2024-03-25T18:14:46.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 -- 172.21.15.87:0/810704453 wait complete. 2024-03-25T18:14:46.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:46.761+0000 7f1167606640 1 librados: shutdown 2024-03-25T18:14:46.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:48.797 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:48.795+0000 7fd32addc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:14:48.828 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:14:48.827+0000 7fd32addc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:14:51.571 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:06.768 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.634 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:06.831 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.667 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:06.845 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.698 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:06.872 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.736 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:06.914 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.741 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:06.938 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.768 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:06.941 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:51.800 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:06.997 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.006 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:07.205 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.032 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-25T18:14:52.026+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.066 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:07.265 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.092 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:07.273 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.170 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:07.350 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.180 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-25T18:14:52.173+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.312 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-25T18:14:52.306+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.372 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-25T18:14:52.365+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.492 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:07.670 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.493 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:07.676 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.512 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:07.692 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.552 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:07.735 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.572 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:07.752 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-03-25T18:14:52.632+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.658 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:07.855 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-03-25T18:14:52.700+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:52.785 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:07.959 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.834 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:08.031 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:52.863 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:08.041 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.024 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:08.222 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.106 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:08.305 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-03-25T18:14:53.115+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:53.153 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:08.327 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.161 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-03-25T18:14:53.153+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:53.174 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:08.355 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.208 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:08.405 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.219 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:08.397 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.336 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:08.533 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.482 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:08.681 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.508 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:08.682 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.523 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:08.701 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.549 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:08.730 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.583 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:53.584 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:08.780 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.594 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:08.777 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.604 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:08.801 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.627 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:08.806 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.670+0000 7f963f8ed640 10 monclient: get_monmap_and_config 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.670+0000 7f963f8ed640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.670+0000 7f963f8ed640 10 monclient: monmap: 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.671 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:14:53.671802+0000 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:14:53.671802+0000 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:14:53.672 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.671+0000 7f963f8ed640 1 Processor -- start 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.671+0000 7f963f8ed640 1 -- start start 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.671+0000 7f963f8ed640 10 monclient: init 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:53.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f963806c380 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f963806d670 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f96380af8f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963806c9a0 con 0x7f963806c380 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380ada90 con 0x7f963806d670 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380b0d10 con 0x7f96380af8f0 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963f8ed640 10 monclient(hunting): authenticate will time out at 7808.967140s 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963ce88640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f963806d670 0x7f96380ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:43432/0 (socket says 172.21.15.87:43432) 2024-03-25T18:14:53.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.672+0000 7f963ce88640 1 -- 172.21.15.87:0/1892235836 learned_addr learned my addr 172.21.15.87:0/1892235836 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992482415 0 0) 0x7f96380ada90 con 0x7f963806d670 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624003520 con 0x7f963806d670 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512246980 0 0) 0x7f96380b0d10 con 0x7f96380af8f0 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96380ada90 con 0x7f96380af8f0 2024-03-25T18:14:53.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301202055 0 0) 0x7f963806c9a0 con 0x7f963806c380 2024-03-25T18:14:53.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:14:53.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.675+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96380b0d10 con 0x7f963806c380 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234760437 0 0) 0x7f9624003520 con 0x7f963806d670 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 >> v1:172.21.15.87:6790/0 conn(0x7f96380af8f0 legacy=0x7f96380aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 >> v1:172.21.15.87:6789/0 conn(0x7f963806c380 legacy=0x7f96380b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 monclient: found mon.noname-c 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380add00 con 0x7f963806d670 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9628001690 con 0x7f963806d670 2024-03-25T18:14:53.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f962ffff640 10 monclient: dump: 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f963f8ed640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:14:53.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.678+0000 7f963f8ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.679+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96280017f0 con 0x7f963806d670 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f962ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f962ffff640 1 -- 172.21.15.87:0/1892235836 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9628001b80 con 0x7f963806d670 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 4 set_mon_vals no callback set 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f962ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f962ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f962ffff640 10 monclient: dump: 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.682 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 10 monclient: shutdown 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 1 -- 172.21.15.87:0/1892235836 >> v1:172.21.15.171:6789/0 conn(0x7f963806d670 legacy=0x7f96380ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 1 -- 172.21.15.87:0/1892235836 shutdown_connections 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 1 -- 172.21.15.87:0/1892235836 >> 172.21.15.87:0/1892235836 conn(0x7f96380b5c00 msgr2=0x7f96380b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 1 -- 172.21.15.87:0/1892235836 shutdown_connections 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.680+0000 7f963f8ed640 1 -- 172.21.15.87:0/1892235836 wait complete. 2024-03-25T18:14:53.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 10 monclient: monmap: 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 librados: starting msgr at 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 librados: starting objecter 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 Processor -- start 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 -- start start 2024-03-25T18:14:53.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 librados: setting wanted keys 2024-03-25T18:14:53.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 1 librados: calling monclient init 2024-03-25T18:14:53.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.681+0000 7f963f8ed640 10 monclient: init 2024-03-25T18:14:53.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f963806c380 addr v1:172.21.15.171:6789/0 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f963806d670 addr v1:172.21.15.87:6790/0 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f96380af8f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963813c8e0 con 0x7f96380af8f0 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638145430 con 0x7f963806c380 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638144770 con 0x7f963806d670 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963f8ed640 10 monclient(hunting): authenticate will time out at 7808.977163s 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963de8a640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f96380af8f0 0x7f963813c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:56852/0 (socket says 172.21.15.87:56852) 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.682+0000 7f963de8a640 1 -- 172.21.15.87:0/2187335395 learned_addr learned my addr 172.21.15.87:0/2187335395 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter ms_handle_connect 0x7f96380af8f0 2024-03-25T18:14:53.687 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter ms_handle_connect 0x7f963806d670 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter ms_handle_connect 0x7f963806c380 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.683+0000 7f962dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832698889 0 0) 0x7f9638144770 con 0x7f963806d670 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963806d670 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528284008 0 0) 0x7f963813c8e0 con 0x7f96380af8f0 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.684+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9638144770 con 0x7f96380af8f0 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447269698 0 0) 0x7f9638145430 con 0x7f963806c380 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:14:53.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.685+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963813c8e0 con 0x7f963806c380 2024-03-25T18:14:53.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168231921 0 0) 0x7f961c004eb0 con 0x7f963806d670 2024-03-25T18:14:53.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9638145430 con 0x7f963806d670 2024-03-25T18:14:53.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9628001b00 con 0x7f963806d670 2024-03-25T18:14:53.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): dump: 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2023668678 0 0) 0x7f963813c8e0 con 0x7f963806c380 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963806c380 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9630001bf0 con 0x7f963806c380 2024-03-25T18:14:53.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): dump: 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396877174 0 0) 0x7f9638144770 con 0x7f96380af8f0 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963813c8e0 con 0x7f96380af8f0 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9634001850 con 0x7f96380af8f0 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.689+0000 7f962dffb640 10 monclient(hunting): dump: 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.693 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3126659992 0 0) 0x7f961c004eb0 con 0x7f963806c380 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 >> v1:172.21.15.87:6790/0 conn(0x7f963806d670 legacy=0x7f963813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 >> v1:172.21.15.87:6789/0 conn(0x7f96380af8f0 legacy=0x7f963813c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 1 monclient: found mon.b 2024-03-25T18:14:53.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638144a50 con 0x7f963806c380 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f962dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 10 monclient: _renew_subs 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9638144d30 con 0x7f963806c380 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 10 monclient: _renew_subs 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963814e0c0 con 0x7f963806c380 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.692+0000 7f963f8ed640 1 librados: init done 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.693+0000 7f963f8ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.693+0000 7f963f8ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.693+0000 7f963f8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:53.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.693+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96380627b0 con 0x7f963806c380 2024-03-25T18:14:53.800 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:08.999 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.800 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:08.974 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.837 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:09.015 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.878 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:09.075 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.878 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:09.075 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.908 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.089 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9630002040 con 0x7f963806c380 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9630003b30 con 0x7f963806c380 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:53.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962dffb640 10 monclient: dump: 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:14:53.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.936+0000 7f962effd640 4 set_mon_vals no callback set 2024-03-25T18:14:53.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.937+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f9630019b00 con 0x7f963806c380 2024-03-25T18:14:53.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.937+0000 7f963ce88640 10 monclient: get_auth_request con 0x7f961c06e150 auth_method 0 2024-03-25T18:14:53.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.937+0000 7f962dffb640 10 client.4207.objecter ms_handle_connect 0x7f961c06e150 2024-03-25T18:14:53.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.942+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f9630053800 con 0x7f963806c380 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.942+0000 7f962dffb640 10 client.4207.objecter ms_dispatch 0x7f9638000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.942+0000 7f962dffb640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.942+0000 7f962dffb640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.942+0000 7f962dffb640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.943+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f963004f140 con 0x7f963806c380 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.943+0000 7f962dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:53.943+0000 7f962dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:53.981 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.164 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.001 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.180 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.069 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.268 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.093 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:09.267 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.123+0000 7f963f8ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.123+0000 7f963f8ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.123+0000 7f963f8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:14:54.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.123+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f963806d160 con 0x7f963806c380 2024-03-25T18:14:54.128 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:09.306 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.146 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:09.343 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.173+0000 7f962dffb640 1 -- 172.21.15.87:0/2187335395 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f9630051240 con 0x7f963806c380 2024-03-25T18:14:54.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.173+0000 7f962dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.173+0000 7f962dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:54.174 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:14:54.174 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 10 librados: watch_flush enter 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 10 librados: watch_flush exit 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 10 client.4207.objecter successfully canceled tick 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 >> v1:172.21.15.171:6800/3417775508 conn(0x7f961c06e150 legacy=0x7f961c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 10 monclient: shutdown 2024-03-25T18:14:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 >> v1:172.21.15.171:6789/0 conn(0x7f963806c380 legacy=0x7f963814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 shutdown_connections 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 >> 172.21.15.87:0/2187335395 conn(0x7f96380b5c00 msgr2=0x7f96381485b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 shutdown_connections 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 -- 172.21.15.87:0/2187335395 wait complete. 2024-03-25T18:14:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:14:54.175+0000 7f963f8ed640 1 librados: shutdown 2024-03-25T18:14:54.180 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:09.377 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:54.206 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.387 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.298 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.481 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.339 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.538 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.370 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.550 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.387 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:09.560 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.422 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:09.619 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.426 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:09.604 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.471 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:09.668 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.493 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.674 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.563 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:09.746 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.603 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:09.802 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.658 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:09.837 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.677 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:09.851 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.697 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:09.894 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.729 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:09.907 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.769 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:09.965 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.781 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:09.961 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.830 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.013 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.871 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.070 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.944 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.124 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.968 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:10.165 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:54.973 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:10.147 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.021 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:10.199 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.058 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:10.255 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.068 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.248 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.095 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.278 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.139 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.338 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.236 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.415 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.275 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:10.449 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.362 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.542 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.364 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:10.561 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.369 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.552 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.405 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:10.604 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.437 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:10.634 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.491 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:10.688 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.523 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.702 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.556 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:10.733 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.610 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:10.788 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.635 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:10.818 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.657 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:10.838 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.795 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:10.969 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.809 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:10.989 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.850 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:11.024 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.876 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:11.075 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.887 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:11.084 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.906 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.089 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.929 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:11.128 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:55.941 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:11.138 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.100 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.280 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.208 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:11.389 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.263 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:11.443 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.402 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.584 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.487 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:11.670 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.628 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.807 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:56.730 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:11.909 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:00.586 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:15.783 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:00.922 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:16.100 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:00.993 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.099+0000 7fe77e5cf640 10 monclient: get_monmap_and_config 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.099+0000 7fe77e5cf640 10 monclient: build_initial_monmap 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.099+0000 7fe77e5cf640 10 monclient: monmap: 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:01.100323+0000 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:01.100323+0000 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:01.100 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.101 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:01.101 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:01.101 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:01.101 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 1 Processor -- start 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 1 -- start start 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient: init 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe7780b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:01.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe7780b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe77814be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780af940 con 0x7fe7780b1730 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780b0a70 con 0x7fe7780b4d60 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780b0d50 con 0x7fe77814be90 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.100+0000 7fe77e5cf640 10 monclient(hunting): authenticate will time out at 7816.395762s 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.101+0000 7fe7777fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fe7780b1730 0x7fe7780b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54894/0 (socket says 172.21.15.87:54894) 2024-03-25T18:15:01.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.101+0000 7fe7777fe640 1 -- 172.21.15.87:0/252520593 learned_addr learned my addr 172.21.15.87:0/252520593 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262106689 0 0) 0x7fe7780af940 con 0x7fe7780b1730 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:15:01.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.103+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760003520 con 0x7fe7780b1730 2024-03-25T18:15:01.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038662397 0 0) 0x7fe7780b0d50 con 0x7fe77814be90 2024-03-25T18:15:01.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7780af940 con 0x7fe77814be90 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423952747 0 0) 0x7fe7780b0a70 con 0x7fe7780b4d60 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:15:01.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.105+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7780b0d50 con 0x7fe7780b4d60 2024-03-25T18:15:01.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982633101 0 0) 0x7fe760003520 con 0x7fe7780b1730 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 >> v1:172.21.15.87:6790/0 conn(0x7fe77814be90 legacy=0x7fe77814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 >> v1:172.21.15.171:6789/0 conn(0x7fe7780b4d60 legacy=0x7fe77813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 monclient: found mon.noname-a 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7780b1030 con 0x7fe7780b1730 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe76c001690 con 0x7fe7780b1730 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:01.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe776ffd640 10 monclient: dump: 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe77e5cf640 5 monclient: authenticate success, global_id 4206 2024-03-25T18:15:01.109 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.107+0000 7fe77e5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:01.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe76c0017f0 con 0x7fe7780b1730 2024-03-25T18:15:01.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:01.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 1 -- 172.21.15.87:0/252520593 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe76c001c00 con 0x7fe7780b1730 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.108+0000 7fe776ffd640 10 monclient: dump: 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:01.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 4 set_mon_vals no callback set 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 10 monclient: shutdown 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 1 -- 172.21.15.87:0/252520593 >> v1:172.21.15.87:6789/0 conn(0x7fe7780b1730 legacy=0x7fe7780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 1 -- 172.21.15.87:0/252520593 shutdown_connections 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.109+0000 7fe77e5cf640 1 -- 172.21.15.87:0/252520593 >> 172.21.15.87:0/252520593 conn(0x7fe7780617a0 msgr2=0x7fe778061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.110+0000 7fe77e5cf640 1 -- 172.21.15.87:0/252520593 shutdown_connections 2024-03-25T18:15:01.112 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.110+0000 7fe77e5cf640 1 -- 172.21.15.87:0/252520593 wait complete. 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.111+0000 7fe77e5cf640 10 monclient: build_initial_monmap 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.111+0000 7fe77e5cf640 10 monclient: monmap: 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:01.113 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.111+0000 7fe77e5cf640 1 librados: starting msgr at 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.111+0000 7fe77e5cf640 1 librados: starting objecter 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.112+0000 7fe77e5cf640 1 Processor -- start 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.112+0000 7fe77e5cf640 1 -- start start 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.112+0000 7fe77e5cf640 1 librados: setting wanted keys 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.112+0000 7fe77e5cf640 1 librados: calling monclient init 2024-03-25T18:15:01.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.112+0000 7fe77e5cf640 10 monclient: init 2024-03-25T18:15:01.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.113+0000 7fe77e5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:01.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.113+0000 7fe77e5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:01.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.113+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe7780b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:01.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.113+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe7780b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe77814be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe778141f10 con 0x7fe7780b1730 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe777fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fe7780b1730 0x7fe7781405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54896/0 (socket says 172.21.15.87:54896) 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe777fff640 1 -- 172.21.15.87:0/3271552206 learned_addr learned my addr 172.21.15.87:0/3271552206 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe778140da0 con 0x7fe77814be90 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe778138860 con 0x7fe7780b4d60 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe77e5cf640 10 monclient(hunting): authenticate will time out at 7816.409364s 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter ms_handle_connect 0x7fe7780b1730 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter ms_handle_connect 0x7fe7780b4d60 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter ms_handle_connect 0x7fe77814be90 2024-03-25T18:15:01.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.114+0000 7fe774ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280019927 0 0) 0x7fe778141f10 con 0x7fe7780b1730 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:15:01.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe764004eb0 con 0x7fe7780b1730 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718296133 0 0) 0x7fe778140da0 con 0x7fe77814be90 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778141f10 con 0x7fe77814be90 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373736090 0 0) 0x7fe778138860 con 0x7fe7780b4d60 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:15:01.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.117+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778140da0 con 0x7fe7780b4d60 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197139292 0 0) 0x7fe764004eb0 con 0x7fe7780b1730 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe778138860 con 0x7fe7780b1730 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe768001a10 con 0x7fe7780b1730 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.124 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.122+0000 7fe774ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24106734 0 0) 0x7fe778140da0 con 0x7fe7780b4d60 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe764004eb0 con 0x7fe7780b4d60 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe76c001b00 con 0x7fe7780b4d60 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.123+0000 7fe774ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.125 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518499438 0 0) 0x7fe778141f10 con 0x7fe77814be90 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe778140da0 con 0x7fe77814be90 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe770001b50 con 0x7fe77814be90 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.124+0000 7fe774ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.127 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.128 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.128 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1736942919 0 0) 0x7fe764004eb0 con 0x7fe7780b4d60 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 >> v1:172.21.15.171:6789/0 conn(0x7fe77814be90 legacy=0x7fe7781e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 >> v1:172.21.15.87:6789/0 conn(0x7fe7780b1730 legacy=0x7fe7781405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 1 monclient: found mon.c 2024-03-25T18:15:01.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe778137f50 con 0x7fe7780b4d60 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 10 monclient: _renew_subs 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7781d8080 con 0x7fe7780b4d60 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 10 monclient: _renew_subs 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7781d8510 con 0x7fe7780b4d60 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.127+0000 7fe77e5cf640 1 librados: init done 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.128+0000 7fe77e5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.128+0000 7fe77e5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.128+0000 7fe77e5cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:01.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.128+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7781381a0 con 0x7fe7780b4d60 2024-03-25T18:15:01.136 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:16.333 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:01.170 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:16.343 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:01.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.172+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe76c001c60 con 0x7fe7780b4d60 2024-03-25T18:15:01.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.172+0000 7fe774ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:01.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.172+0000 7fe7767fc640 4 set_mon_vals no callback set 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.213+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe76c004c30 con 0x7fe7780b4d60 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.213+0000 7fe774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.213+0000 7fe774ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.213+0000 7fe774ff9640 10 monclient: dump: 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:01.214 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:01.215 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:01.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.465+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fe76c01ac00 con 0x7fe7780b4d60 2024-03-25T18:15:01.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.466+0000 7fe777fff640 10 monclient: get_auth_request con 0x7fe76406e0b0 auth_method 0 2024-03-25T18:15:01.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.466+0000 7fe774ff9640 10 client.4214.objecter ms_handle_connect 0x7fe76406e0b0 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fe76c054890 con 0x7fe7780b4d60 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 10 client.4214.objecter ms_dispatch 0x7fe778000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe76c0501d0 con 0x7fe7780b4d60 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.477+0000 7fe774ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:01.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.717+0000 7fe77e5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.717+0000 7fe77e5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.717+0000 7fe77e5cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:01.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.717+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7780627b0 con 0x7fe7780b4d60 2024-03-25T18:15:01.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.781+0000 7fe774ff9640 1 -- 172.21.15.87:0/3271552206 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fe76c0522d0 con 0x7fe7780b4d60 2024-03-25T18:15:01.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.781+0000 7fe774ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.781+0000 7fe774ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:01.782 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:01.782 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.783+0000 7fe77e5cf640 10 librados: watch_flush enter 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.783+0000 7fe77e5cf640 10 librados: watch_flush exit 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.783+0000 7fe77e5cf640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.783+0000 7fe77e5cf640 10 client.4214.objecter successfully canceled tick 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.783+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 >> v1:172.21.15.171:6800/3417775508 conn(0x7fe76406e0b0 legacy=0x7fe764070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 10 monclient: shutdown 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 >> v1:172.21.15.87:6790/0 conn(0x7fe7780b4d60 legacy=0x7fe7781d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 shutdown_connections 2024-03-25T18:15:01.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 >> 172.21.15.87:0/3271552206 conn(0x7fe7780617a0 msgr2=0x7fe7780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 shutdown_connections 2024-03-25T18:15:01.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 -- 172.21.15.87:0/3271552206 wait complete. 2024-03-25T18:15:01.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:01.784+0000 7fe77e5cf640 1 librados: shutdown 2024-03-25T18:15:01.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:01.942 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:17.123 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:01.946 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:17.145 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:02.288 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:17.471 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:02.569 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:17.749 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.125 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:21.322 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.131 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:21.328 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.380 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:21.554 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.384 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:21.558 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.536 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:21.714 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:06.541 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:21.718 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.199 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:22.396 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.203 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:22.400 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.360 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:22.559 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.364 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:22.563 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.510 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:22.690 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:07.515 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:22.696 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:08.066 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:23.249 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:08.070 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:23.253 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:08.534 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:23.714 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:08.539 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:23.719 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:08.606 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:08.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.693+0000 7f0d51b2b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:08.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.693+0000 7f0d51b2b640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.693+0000 7f0d51b2b640 10 monclient: monmap: 2024-03-25T18:15:08.695 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:08.695 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:08.695177+0000 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:08.695177+0000 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.694+0000 7f0d51b2b640 1 Processor -- start 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.694+0000 7f0d51b2b640 1 -- start start 2024-03-25T18:15:08.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.694+0000 7f0d51b2b640 10 monclient: init 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c0b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c14be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0af940 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0b0a70 con 0x7f0d4c0b1730 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0b0d50 con 0x7f0d4c14be90 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d51b2b640 10 monclient(hunting): authenticate will time out at 7823.991326s 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d4affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0d4c0b1730 0x7f0d4c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:47678/0 (socket says 172.21.15.87:47678) 2024-03-25T18:15:08.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.695+0000 7f0d4affd640 1 -- 172.21.15.87:0/65385747 learned_addr learned my addr 172.21.15.87:0/65385747 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717466347 0 0) 0x7f0d4c0af940 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003520 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710682872 0 0) 0x7f0d4c0b0a70 con 0x7f0d4c0b1730 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.698+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c0af940 con 0x7f0d4c0b1730 2024-03-25T18:15:08.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223051040 0 0) 0x7f0d4c0b0d50 con 0x7f0d4c14be90 2024-03-25T18:15:08.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:15:08.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.699+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c0b0a70 con 0x7f0d4c14be90 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838573359 0 0) 0x7f0d4c0af940 con 0x7f0d4c0b1730 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 >> v1:172.21.15.87:6790/0 conn(0x7f0d4c14be90 legacy=0x7f0d4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 >> v1:172.21.15.87:6789/0 conn(0x7f0d4c0b4d60 legacy=0x7f0d4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c0b1030 con 0x7f0d4c0b1730 2024-03-25T18:15:08.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.702+0000 7f0d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d400015a0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d4a7fc640 10 monclient: dump: 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.706 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.707 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d51b2b640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:15:08.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.703+0000 7f0d51b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d400017c0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 1 -- 172.21.15.87:0/65385747 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d40001b50 con 0x7f0d4c0b1730 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d4a7fc640 10 monclient: dump: 2024-03-25T18:15:08.709 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d51b2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.704+0000 7f0d51b2b640 4 set_mon_vals no callback set 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.705+0000 7f0d51b2b640 10 monclient: shutdown 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.705+0000 7f0d51b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.705+0000 7f0d51b2b640 1 -- 172.21.15.87:0/65385747 >> v1:172.21.15.171:6789/0 conn(0x7f0d4c0b1730 legacy=0x7f0d4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.705+0000 7f0d51b2b640 1 -- 172.21.15.87:0/65385747 shutdown_connections 2024-03-25T18:15:08.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.705+0000 7f0d51b2b640 1 -- 172.21.15.87:0/65385747 >> 172.21.15.87:0/65385747 conn(0x7f0d4c0617a0 msgr2=0x7f0d4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.706+0000 7f0d51b2b640 1 -- 172.21.15.87:0/65385747 shutdown_connections 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.706+0000 7f0d51b2b640 1 -- 172.21.15.87:0/65385747 wait complete. 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 10 monclient: monmap: 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 librados: starting msgr at 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 librados: starting objecter 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 Processor -- start 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 -- start start 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 librados: setting wanted keys 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 1 librados: calling monclient init 2024-03-25T18:15:08.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.707+0000 7f0d51b2b640 10 monclient: init 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c0b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c0b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.708+0000 7f0d51b2b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c142a30 con 0x7f0d4c14be90 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d51b2b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c141bc0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d51b2b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c141e30 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d51b2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d51b2b640 10 monclient(hunting): authenticate will time out at 7824.004906s 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d4b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0d4c0b1730 0x7f0d4c142ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:47682/0 (socket says 172.21.15.87:47682) 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d4b7fe640 1 -- 172.21.15.87:0/2429845207 learned_addr learned my addr 172.21.15.87:0/2429845207 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d33fff640 10 client.?.objecter ms_handle_connect 0x7f0d4c0b1730 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d33fff640 10 client.?.objecter ms_handle_connect 0x7f0d4c14be90 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.709+0000 7f0d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.710+0000 7f0d33fff640 10 client.?.objecter ms_handle_connect 0x7f0d4c0b4d60 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.710+0000 7f0d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965649438 0 0) 0x7f0d4c141bc0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.711+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d38004eb0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524172058 0 0) 0x7f0d4c142a30 con 0x7f0d4c14be90 2024-03-25T18:15:08.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c141bc0 con 0x7f0d4c14be90 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558243087 0 0) 0x7f0d4c141e30 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:15:08.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.712+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c142a30 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585872086 0 0) 0x7f0d38004eb0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c141e30 con 0x7f0d4c0b1730 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d3c001990 con 0x7f0d4c0b1730 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 10 monclient(hunting): dump: 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.719 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.720 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.720 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.716+0000 7f0d33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007975790 0 0) 0x7f0d4c142a30 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d38004eb0 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d400014e0 con 0x7f0d4c0b4d60 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.721 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): dump: 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125846999 0 0) 0x7f0d4c141bc0 con 0x7f0d4c14be90 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c142a30 con 0x7f0d4c14be90 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d44001bd0 con 0x7f0d4c14be90 2024-03-25T18:15:08.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.719+0000 7f0d33fff640 10 monclient(hunting): dump: 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.723 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4139050145 0 0) 0x7f0d4c141e30 con 0x7f0d4c0b1730 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 >> v1:172.21.15.87:6790/0 conn(0x7f0d4c0b4d60 legacy=0x7f0d4c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 >> v1:172.21.15.87:6789/0 conn(0x7f0d4c14be90 legacy=0x7f0d4c135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 1 monclient: found mon.b 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c135770 con 0x7f0d4c0b1730 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:15:08.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 10 monclient: _renew_subs 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c143690 con 0x7f0d4c0b1730 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 10 monclient: _renew_subs 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c138ef0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.720+0000 7f0d51b2b640 1 librados: init done 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.722+0000 7f0d51b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.722+0000 7f0d51b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.722+0000 7f0d51b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:08.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.722+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c0afbd0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d3c003610 con 0x7f0d4c0b1730 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d3c0039e0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d33fff640 10 monclient: dump: 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:08.745 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:08.746 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:08.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.743+0000 7f0d49ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:08.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.744+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f0d3c003c00 con 0x7f0d4c0b1730 2024-03-25T18:15:08.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.745+0000 7f0d4affd640 10 monclient: get_auth_request con 0x7f0d3806e180 auth_method 0 2024-03-25T18:15:08.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.745+0000 7f0d33fff640 10 client.4219.objecter ms_handle_connect 0x7f0d3806e180 2024-03-25T18:15:08.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.750+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f0d3c0547b0 con 0x7f0d4c0b1730 2024-03-25T18:15:08.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.750+0000 7f0d33fff640 10 client.4219.objecter ms_dispatch 0x7f0d4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:08.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.750+0000 7f0d33fff640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:08.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.750+0000 7f0d33fff640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:08.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:08.750+0000 7f0d33fff640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T18:15:09.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.077+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d3c0502c0 con 0x7f0d4c0b1730 2024-03-25T18:15:09.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.077+0000 7f0d33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:09.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.077+0000 7f0d33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:09.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.419+0000 7f0d51b2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.419+0000 7f0d51b2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.419+0000 7f0d51b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:09.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.419+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d4c1391a0 con 0x7f0d4c0b1730 2024-03-25T18:15:09.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.438+0000 7f0d33fff640 1 -- 172.21.15.87:0/2429845207 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f0d3c0521f0 con 0x7f0d4c0b1730 2024-03-25T18:15:09.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.438+0000 7f0d33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.438+0000 7f0d33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:09.440 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:09.440 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.440+0000 7f0d51b2b640 10 librados: watch_flush enter 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 10 librados: watch_flush exit 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 10 client.4219.objecter successfully canceled tick 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 >> v1:172.21.15.171:6800/3417775508 conn(0x7f0d3806e180 legacy=0x7f0d38070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 10 monclient: shutdown 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 >> v1:172.21.15.171:6789/0 conn(0x7f0d4c0b1730 legacy=0x7f0d4c142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 shutdown_connections 2024-03-25T18:15:09.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 >> 172.21.15.87:0/2429845207 conn(0x7f0d4c0617a0 msgr2=0x7f0d4c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:09.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 shutdown_connections 2024-03-25T18:15:09.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 -- 172.21.15.87:0/2429845207 wait complete. 2024-03-25T18:15:09.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:09.441+0000 7f0d51b2b640 1 librados: shutdown 2024-03-25T18:15:09.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:10.617 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:00:25.814 35347== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:10.836 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:00:26.014 35349== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:10.838 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:00:26.012 35350== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:11.422 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:00:26.621 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:11.451 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:00:26.647 35348== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:11.723 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:00:26.903 35206== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:12.521 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:00:27.704 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:12.907 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:00:28.086 35207== Warning: unimplemented fcntl command: 1038 2024-03-25T18:15:16.261 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:16.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.354+0000 7f6a796c9640 10 monclient: get_monmap_and_config 2024-03-25T18:15:16.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.354+0000 7f6a796c9640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.354+0000 7f6a796c9640 10 monclient: monmap: 2024-03-25T18:15:16.355 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:16.355 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:16.355572+0000 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:16.355572+0000 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:16.356 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 1 Processor -- start 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 1 -- start start 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 10 monclient: init 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-b con 0x7f6a740b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.355+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-a con 0x7f6a740b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-c con 0x7f6a7414be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740af940 con 0x7f6a740b1730 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740b0a70 con 0x7f6a7414be90 2024-03-25T18:15:16.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740b0d50 con 0x7f6a740b4d60 2024-03-25T18:15:16.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a796c9640 10 monclient(hunting): authenticate will time out at 7831.651045s 2024-03-25T18:15:16.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a72ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f6a740b4d60 0x7f6a7413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:43440/0 (socket says 172.21.15.87:43440) 2024-03-25T18:15:16.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a737fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6a7414be90 0x7f6a7414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:47698/0 (socket says 172.21.15.87:47698) 2024-03-25T18:15:16.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.356+0000 7f6a72ffd640 1 -- 172.21.15.87:0/3497055016 learned_addr learned my addr 172.21.15.87:0/3497055016 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:16.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566377698 0 0) 0x7f6a740af940 con 0x7f6a740b1730 2024-03-25T18:15:16.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a5c003520 con 0x7f6a740b1730 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525185309 0 0) 0x7f6a740b0d50 con 0x7f6a740b4d60 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.359+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a740af940 con 0x7f6a740b4d60 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201259510 0 0) 0x7f6a740b0a70 con 0x7f6a7414be90 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:15:16.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.361+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a740b0d50 con 0x7f6a7414be90 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2622215042 0 0) 0x7f6a5c003520 con 0x7f6a740b1730 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 >> v1:172.21.15.87:6790/0 conn(0x7f6a740b4d60 legacy=0x7f6a7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 >> v1:172.21.15.171:6789/0 conn(0x7f6a7414be90 legacy=0x7f6a7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 1 monclient: found mon.noname-a 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740b1030 con 0x7f6a740b1730 2024-03-25T18:15:16.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a796c9640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.363+0000 7f6a796c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.364+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a68001650 con 0x7f6a740b1730 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.364+0000 7f6a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.364+0000 7f6a71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.364+0000 7f6a71ffb640 10 monclient: dump: 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.366 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.367 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.367 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.367 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.364+0000 7f6a796c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:16.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.366+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a680018a0 con 0x7f6a740b1730 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.366+0000 7f6a71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.366+0000 7f6a71ffb640 1 -- 172.21.15.87:0/3497055016 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a68001c30 con 0x7f6a740b1730 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.366+0000 7f6a796c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.366+0000 7f6a796c9640 4 set_mon_vals no callback set 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a796c9640 10 monclient: shutdown 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a71ffb640 10 monclient: dump: 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.369 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a796c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.367+0000 7f6a796c9640 1 -- 172.21.15.87:0/3497055016 >> v1:172.21.15.87:6789/0 conn(0x7f6a740b1730 legacy=0x7f6a740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.368+0000 7f6a796c9640 1 -- 172.21.15.87:0/3497055016 shutdown_connections 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.368+0000 7f6a796c9640 1 -- 172.21.15.87:0/3497055016 >> 172.21.15.87:0/3497055016 conn(0x7f6a740617a0 msgr2=0x7f6a74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.368+0000 7f6a796c9640 1 -- 172.21.15.87:0/3497055016 shutdown_connections 2024-03-25T18:15:16.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.368+0000 7f6a796c9640 1 -- 172.21.15.87:0/3497055016 wait complete. 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.369+0000 7f6a796c9640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.369+0000 7f6a796c9640 10 monclient: monmap: 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:16.371 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 librados: starting msgr at 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 librados: starting objecter 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 Processor -- start 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 -- start start 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 librados: setting wanted keys 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 1 librados: calling monclient init 2024-03-25T18:15:16.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.370+0000 7f6a796c9640 10 monclient: init 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.371+0000 7f6a796c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.371+0000 7f6a796c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.371+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-c con 0x7f6a740b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.371+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-a con 0x7f6a740b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.371+0000 7f6a796c9640 10 monclient(hunting): picked mon.noname-b con 0x7f6a7414be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:16.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a796c9640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a74140e50 con 0x7f6a740b4d60 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a796c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a74140930 con 0x7f6a7414be90 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a796c9640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a74138760 con 0x7f6a740b1730 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a796c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a727fc640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6a740b4d60 0x7f6a741dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57494/0 (socket says 172.21.15.87:57494) 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a727fc640 1 -- 172.21.15.87:0/3067494493 learned_addr learned my addr 172.21.15.87:0/3067494493 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a796c9640 10 monclient(hunting): authenticate will time out at 7831.667313s 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6a740b4d60 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6a740b1730 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.372+0000 7f6a5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.373+0000 7f6a5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6a7414be90 2024-03-25T18:15:16.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.373+0000 7f6a5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063514022 0 0) 0x7f6a74140e50 con 0x7f6a740b4d60 2024-03-25T18:15:16.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.374+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60004eb0 con 0x7f6a740b4d60 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472940417 0 0) 0x7f6a74140930 con 0x7f6a7414be90 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.375+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a74140e50 con 0x7f6a7414be90 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615641014 0 0) 0x7f6a74138760 con 0x7f6a740b1730 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:15:16.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.376+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a74140930 con 0x7f6a740b1730 2024-03-25T18:15:16.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587507526 0 0) 0x7f6a60004eb0 con 0x7f6a740b4d60 2024-03-25T18:15:16.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a74138760 con 0x7f6a740b4d60 2024-03-25T18:15:16.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a68001c70 con 0x7f6a740b4d60 2024-03-25T18:15:16.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.381+0000 7f6a5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2630243501 0 0) 0x7f6a74140e50 con 0x7f6a7414be90 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a60004eb0 con 0x7f6a7414be90 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a6c001b90 con 0x7f6a7414be90 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.385 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 307089316 0 0) 0x7f6a74140930 con 0x7f6a740b1730 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a74140e50 con 0x7f6a740b1730 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a640018b0 con 0x7f6a740b1730 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.383+0000 7f6a5b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.386 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1519267485 0 0) 0x7f6a74138760 con 0x7f6a740b4d60 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 >> v1:172.21.15.87:6790/0 conn(0x7f6a740b1730 legacy=0x7f6a74131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 >> v1:172.21.15.171:6789/0 conn(0x7f6a7414be90 legacy=0x7f6a741e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 1 monclient: found mon.a 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a741388f0 con 0x7f6a740b4d60 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 10 monclient: _renew_subs 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a74137e90 con 0x7f6a740b4d60 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 10 monclient: _renew_subs 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a74132800 con 0x7f6a740b4d60 2024-03-25T18:15:16.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.386+0000 7f6a796c9640 1 librados: init done 2024-03-25T18:15:16.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.387+0000 7f6a796c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.387+0000 7f6a796c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.387+0000 7f6a796c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.387+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a741401c0 con 0x7f6a740b4d60 2024-03-25T18:15:16.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a68001230 con 0x7f6a740b4d60 2024-03-25T18:15:16.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6a68003b00 con 0x7f6a740b4d60 2024-03-25T18:15:16.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a5b7fe640 10 monclient: dump: 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:16.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.604+0000 7f6a717fa640 4 set_mon_vals no callback set 2024-03-25T18:15:16.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.615+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f6a68003c60 con 0x7f6a740b4d60 2024-03-25T18:15:16.616 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.615+0000 7f6a72ffd640 10 monclient: get_auth_request con 0x7f6a6006e0f0 auth_method 0 2024-03-25T18:15:16.616 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.616+0000 7f6a5b7fe640 10 client.4221.objecter ms_handle_connect 0x7f6a6006e0f0 2024-03-25T18:15:16.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.659+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f6a680548f0 con 0x7f6a740b4d60 2024-03-25T18:15:16.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.659+0000 7f6a5b7fe640 10 client.4221.objecter ms_dispatch 0x7f6a74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:16.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.659+0000 7f6a5b7fe640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:16.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.659+0000 7f6a5b7fe640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:16.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.659+0000 7f6a5b7fe640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T18:15:16.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.713+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a68050230 con 0x7f6a740b4d60 2024-03-25T18:15:16.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.713+0000 7f6a5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.713+0000 7f6a5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:16.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.993+0000 7f6a796c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.993+0000 7f6a796c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:16.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.993+0000 7f6a796c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:16.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:16.994+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a740627b0 con 0x7f6a740b4d60 2024-03-25T18:15:17.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.008+0000 7f6a5b7fe640 1 -- 172.21.15.87:0/3067494493 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f6a68052330 con 0x7f6a740b4d60 2024-03-25T18:15:17.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.008+0000 7f6a5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:17.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.008+0000 7f6a5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:17.009 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:17.009 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.010+0000 7f6a796c9640 10 librados: watch_flush enter 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 10 librados: watch_flush exit 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 10 client.4221.objecter successfully canceled tick 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 >> v1:172.21.15.171:6800/3417775508 conn(0x7f6a6006e0f0 legacy=0x7f6a600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 10 monclient: shutdown 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 >> v1:172.21.15.87:6789/0 conn(0x7f6a740b4d60 legacy=0x7f6a741dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 shutdown_connections 2024-03-25T18:15:17.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 >> 172.21.15.87:0/3067494493 conn(0x7f6a740617a0 msgr2=0x7f6a7413c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:17.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 shutdown_connections 2024-03-25T18:15:17.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 -- 172.21.15.87:0/3067494493 wait complete. 2024-03-25T18:15:17.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:17.011+0000 7f6a796c9640 1 librados: shutdown 2024-03-25T18:15:17.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:19.125 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-03-25T18:15:19.121+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:15:19.364 INFO:tasks.ceph.osd.7.smithi171.stderr:2024-03-25T18:15:19.353+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:15:19.372 INFO:tasks.ceph.osd.6.smithi171.stderr:2024-03-25T18:15:19.299+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:15:19.974 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-03-25T18:15:19.971+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:15:20.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2024-03-25T18:15:20.338+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:15:21.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2024-03-25T18:15:21.236+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:15:21.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2024-03-25T18:15:21.943+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:15:22.655 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-03-25T18:15:22.650+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:15:23.828 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.946+0000 7fdfb50a8640 10 monclient: get_monmap_and_config 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.946+0000 7fdfb50a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.946+0000 7fdfb50a8640 10 monclient: monmap: 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:23.947 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:23.947657+0000 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:23.947657+0000 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:23.948 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:23.949 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.947+0000 7fdfb50a8640 1 Processor -- start 2024-03-25T18:15:23.949 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.947+0000 7fdfb50a8640 1 -- start start 2024-03-25T18:15:23.949 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.947+0000 7fdfb50a8640 10 monclient: init 2024-03-25T18:15:23.949 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.949 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb00a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb013f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb01430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0142520 con 0x7fdfb013f710 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00b28a0 con 0x7fdfb01430c0 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00b2a30 con 0x7fdfb00a98f0 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.948+0000 7fdfb50a8640 10 monclient(hunting): authenticate will time out at 7839.243703s 2024-03-25T18:15:23.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.952+0000 7fdfaed9d640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fdfb00a98f0 0x7fdfb00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:56446/0 (socket says 172.21.15.87:56446) 2024-03-25T18:15:23.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.952+0000 7fdfaed9d640 1 -- 172.21.15.87:0/1872932174 learned_addr learned my addr 172.21.15.87:0/1872932174 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438650817 0 0) 0x7fdfb00b2a30 con 0x7fdfb00a98f0 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:15:23.966 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.965+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb00a98f0 2024-03-25T18:15:23.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505718251 0 0) 0x7fdfb00b28a0 con 0x7fdfb01430c0 2024-03-25T18:15:23.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 10 monclient(hunting): _negotiate 2024-03-25T18:15:23.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:23.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:23.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:15:23.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.970+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb00b2a30 con 0x7fdfb01430c0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226556998 0 0) 0x7fdf9c003520 con 0x7fdfb00a98f0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 >> v1:172.21.15.171:6789/0 conn(0x7fdfb01430c0 legacy=0x7fdfb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 >> v1:172.21.15.87:6789/0 conn(0x7fdfb013f710 legacy=0x7fdfb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: _finish_hunting 0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 monclient: found mon.noname-b 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00b44c0 con 0x7fdfb00a98f0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: _finish_auth 0 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: _check_auth_tickets 2024-03-25T18:15:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdfa0001690 con 0x7fdfb00a98f0 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfae59c640 10 monclient: dump: 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfb50a8640 5 monclient: authenticate success, global_id 4253 2024-03-25T18:15:23.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.977+0000 7fdfb50a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfa00017f0 con 0x7fdfb00a98f0 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 1 -- 172.21.15.87:0/1872932174 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdfa0001c00 con 0x7fdfb00a98f0 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfae59c640 10 monclient: dump: 2024-03-25T18:15:23.993 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:23.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfb50a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:23.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.992+0000 7fdfb50a8640 4 set_mon_vals no callback set 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 10 monclient: shutdown 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 1 -- 172.21.15.87:0/1872932174 >> v1:172.21.15.87:6790/0 conn(0x7fdfb00a98f0 legacy=0x7fdfb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 1 -- 172.21.15.87:0/1872932174 shutdown_connections 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 1 -- 172.21.15.87:0/1872932174 >> 172.21.15.87:0/1872932174 conn(0x7fdfb00b7ea0 msgr2=0x7fdfb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 1 -- 172.21.15.87:0/1872932174 shutdown_connections 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.993+0000 7fdfb50a8640 1 -- 172.21.15.87:0/1872932174 wait complete. 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.994+0000 7fdfb50a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:23.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.994+0000 7fdfb50a8640 10 monclient: monmap: 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.994+0000 7fdfb50a8640 1 librados: starting msgr at 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.994+0000 7fdfb50a8640 1 librados: starting objecter 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.994+0000 7fdfb50a8640 1 Processor -- start 2024-03-25T18:15:23.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 -- start start 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 librados: setting wanted keys 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 librados: calling monclient init 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient: init 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb00a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb01430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0137370 con 0x7fdfb00a98f0 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb01352a0 con 0x7fdfb01430c0 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb0131760 con 0x7fdfb013f710 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:23.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:23.995+0000 7fdfb50a8640 10 monclient(hunting): authenticate will time out at 7839.290498s 2024-03-25T18:15:24.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.000+0000 7fdfaed9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fdfb00a98f0 0x7fdfb0131050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:40546/0 (socket says 172.21.15.87:40546) 2024-03-25T18:15:24.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.000+0000 7fdfaed9d640 1 -- 172.21.15.87:0/2161844781 learned_addr learned my addr 172.21.15.87:0/2161844781 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:24.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.001+0000 7fdfa77fe640 10 client.?.objecter ms_handle_connect 0x7fdfb01430c0 2024-03-25T18:15:24.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.001+0000 7fdfa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:24.003 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.002+0000 7fdfa77fe640 10 client.?.objecter ms_handle_connect 0x7fdfb00a98f0 2024-03-25T18:15:24.003 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.002+0000 7fdfa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:24.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 client.?.objecter ms_handle_connect 0x7fdfb013f710 2024-03-25T18:15:24.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:24.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609456053 0 0) 0x7fdfb01352a0 con 0x7fdfb01430c0 2024-03-25T18:15:24.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:24.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:24.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:24.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:15:24.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.013+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf90004eb0 con 0x7fdfb01430c0 2024-03-25T18:15:24.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056358755 0 0) 0x7fdfb0137370 con 0x7fdfb00a98f0 2024-03-25T18:15:24.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:24.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:24.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:24.024 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:15:24.024 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.023+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb01352a0 con 0x7fdfb00a98f0 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255526198 0 0) 0x7fdfb0131760 con 0x7fdfb013f710 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:15:24.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.026+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb0137370 con 0x7fdfb013f710 2024-03-25T18:15:24.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545317703 0 0) 0x7fdfb0137370 con 0x7fdfb013f710 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb0131760 con 0x7fdfb013f710 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdf98001830 con 0x7fdfb013f710 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 10 monclient(hunting): dump: 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:24.039 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:24.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.038+0000 7fdfa77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.043+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033664452 0 0) 0x7fdf90004eb0 con 0x7fdfb01430c0 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb0137370 con 0x7fdfb01430c0 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdfa8001bd0 con 0x7fdfb01430c0 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): dump: 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.045 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605284121 0 0) 0x7fdfb01352a0 con 0x7fdfb00a98f0 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf90004eb0 con 0x7fdfb00a98f0 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdfa0001b40 con 0x7fdfb00a98f0 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.044+0000 7fdfa77fe640 10 monclient(hunting): dump: 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:24.046 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:24.047 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1283114741 0 0) 0x7fdfb0131760 con 0x7fdfb013f710 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 >> v1:172.21.15.171:6789/0 conn(0x7fdfb01430c0 legacy=0x7fdfb01d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 >> v1:172.21.15.87:6789/0 conn(0x7fdfb00a98f0 legacy=0x7fdfb0131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 1 monclient: found mon.c 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.053+0000 7fdfa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfa77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb01318f0 con 0x7fdfb013f710 2024-03-25T18:15:24.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfa77fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfa77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 5 monclient: authenticate success, global_id 4256 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 10 monclient: _renew_subs 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb00b7a40 con 0x7fdfb013f710 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 10 monclient: _renew_subs 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb01360e0 con 0x7fdfb013f710 2024-03-25T18:15:24.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.054+0000 7fdfb50a8640 1 librados: init done 2024-03-25T18:15:24.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.055+0000 7fdfb50a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:24.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.055+0000 7fdfb50a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:24.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.055+0000 7fdfb50a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:24.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.055+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb0135430 con 0x7fdfb013f710 2024-03-25T18:15:24.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf98001d40 con 0x7fdfb013f710 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fdf98003790 con 0x7fdfb013f710 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfa77fe640 10 monclient: dump: 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:24.358 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:24.359 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:24.359 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:24.359 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:24.359 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:24.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.356+0000 7fdfadd9b640 4 set_mon_vals no callback set 2024-03-25T18:15:24.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.357+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fdf980039b0 con 0x7fdfb013f710 2024-03-25T18:15:24.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.358+0000 7fdfaed9d640 10 monclient: get_auth_request con 0x7fdf9006e0b0 auth_method 0 2024-03-25T18:15:24.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.358+0000 7fdfa77fe640 10 client.4256.objecter ms_handle_connect 0x7fdf9006e0b0 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fdf98053590 con 0x7fdfb013f710 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 10 client.4256.objecter ms_dispatch 0x7fdfb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf9804eed0 con 0x7fdfb013f710 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:24.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.364+0000 7fdfa77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:24.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.588+0000 7fdfb50a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.588+0000 7fdfb50a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.588+0000 7fdfb50a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:15:24.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.588+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfb00627b0 con 0x7fdfb013f710 2024-03-25T18:15:24.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.738+0000 7fdfa77fe640 1 -- 172.21.15.87:0/2161844781 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fdf98050fd0 con 0x7fdfb013f710 2024-03-25T18:15:24.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.738+0000 7fdfa77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:24.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.738+0000 7fdfa77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:24.740 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:24.740 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.741+0000 7fdfb50a8640 10 librados: watch_flush enter 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.741+0000 7fdfb50a8640 10 librados: watch_flush exit 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.741+0000 7fdfb50a8640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.741+0000 7fdfb50a8640 10 client.4256.objecter successfully canceled tick 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 >> v1:172.21.15.171:6800/3417775508 conn(0x7fdf9006e0b0 legacy=0x7fdf90070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 10 monclient: shutdown 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 >> v1:172.21.15.87:6790/0 conn(0x7fdfb013f710 legacy=0x7fdfb01d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 shutdown_connections 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 >> 172.21.15.87:0/2161844781 conn(0x7fdfb00b7ea0 msgr2=0x7fdfb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 shutdown_connections 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 -- 172.21.15.87:0/2161844781 wait complete. 2024-03-25T18:15:24.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:24.742+0000 7fdfb50a8640 1 librados: shutdown 2024-03-25T18:15:24.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:31.560 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.678+0000 7fa7de755640 10 monclient: get_monmap_and_config 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.678+0000 7fa7de755640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.678+0000 7fa7de755640 10 monclient: monmap: 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.679 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:31.679740+0000 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:31.679740+0000 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:31.680 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.679+0000 7fa7de755640 1 Processor -- start 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.679+0000 7fa7de755640 1 -- start start 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.679+0000 7fa7de755640 10 monclient: init 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d80a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:31.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d81430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d8142520 con 0x7fa7d81430c0 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d80b28a0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d80b2a30 con 0x7fa7d813f710 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.680+0000 7fa7de755640 10 monclient(hunting): authenticate will time out at 7846.975399s 2024-03-25T18:15:31.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.688+0000 7fa7d7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa7d80a98f0 0x7fa7d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:36498/0 (socket says 172.21.15.87:36498) 2024-03-25T18:15:31.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.688+0000 7fa7d7fff640 1 -- 172.21.15.87:0/3875760298 learned_addr learned my addr 172.21.15.87:0/3875760298 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444652078 0 0) 0x7fa7d8142520 con 0x7fa7d81430c0 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:15:31.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.699+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c0003520 con 0x7fa7d81430c0 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224193875 0 0) 0x7fa7d80b28a0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:15:31.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.703+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8142520 con 0x7fa7d80a98f0 2024-03-25T18:15:31.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 626082148 0 0) 0x7fa7d8142520 con 0x7fa7d80a98f0 2024-03-25T18:15:31.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 >> v1:172.21.15.87:6790/0 conn(0x7fa7d813f710 legacy=0x7fa7d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 >> v1:172.21.15.87:6789/0 conn(0x7fa7d81430c0 legacy=0x7fa7d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d80b44c0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7c8001670 con 0x7fa7d80a98f0 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7d6ffd640 10 monclient: dump: 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.712 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.710+0000 7fa7de755640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:15:31.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.711+0000 7fa7de755640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:31.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c80017d0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7de755640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 1 -- 172.21.15.87:0/3875760298 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7c8001ba0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7de755640 4 set_mon_vals no callback set 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.723+0000 7fa7d6ffd640 10 monclient: dump: 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:31.724 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:31.725 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:31.725 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 10 monclient: shutdown 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 1 -- 172.21.15.87:0/3875760298 >> v1:172.21.15.171:6789/0 conn(0x7fa7d80a98f0 legacy=0x7fa7d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 1 -- 172.21.15.87:0/3875760298 shutdown_connections 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 1 -- 172.21.15.87:0/3875760298 >> 172.21.15.87:0/3875760298 conn(0x7fa7d80b7ea0 msgr2=0x7fa7d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 1 -- 172.21.15.87:0/3875760298 shutdown_connections 2024-03-25T18:15:31.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 1 -- 172.21.15.87:0/3875760298 wait complete. 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.724+0000 7fa7de755640 10 monclient: build_initial_monmap 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 10 monclient: monmap: 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 librados: starting msgr at 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 librados: starting objecter 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 Processor -- start 2024-03-25T18:15:31.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 -- start start 2024-03-25T18:15:31.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 librados: setting wanted keys 2024-03-25T18:15:31.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 1 librados: calling monclient init 2024-03-25T18:15:31.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.725+0000 7fa7de755640 10 monclient: init 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d80a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d81430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d8138720 con 0x7fa7d81430c0 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d80b1770 con 0x7fa7d80a98f0 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d8134690 con 0x7fa7d813f710 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:31.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.726+0000 7fa7de755640 10 monclient(hunting): authenticate will time out at 7847.021427s 2024-03-25T18:15:31.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.737+0000 7fa7d77fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fa7d813f710 0x7fa7d8133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:44694/0 (socket says 172.21.15.87:44694) 2024-03-25T18:15:31.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.737+0000 7fa7d77fe640 1 -- 172.21.15.87:0/1452054185 learned_addr learned my addr 172.21.15.87:0/1452054185 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:31.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.739+0000 7fa7d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7d80a98f0 2024-03-25T18:15:31.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.740+0000 7fa7d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.742+0000 7fa7d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7d813f710 2024-03-25T18:15:31.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.742+0000 7fa7d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.747+0000 7fa7d4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7d81430c0 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.747+0000 7fa7d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593478200 0 0) 0x7fa7d8134690 con 0x7fa7d813f710 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:15:31.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.748+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c4004eb0 con 0x7fa7d813f710 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287055476 0 0) 0x7fa7d8138720 con 0x7fa7d81430c0 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:15:31.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.758+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8134690 con 0x7fa7d81430c0 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725044202 0 0) 0x7fa7d80b1770 con 0x7fa7d80a98f0 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:15:31.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.760+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8138720 con 0x7fa7d80a98f0 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3955349454 0 0) 0x7fa7c4004eb0 con 0x7fa7d813f710 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d80b1770 con 0x7fa7d813f710 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7cc001a30 con 0x7fa7d813f710 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:31.762 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.761+0000 7fa7d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306809146 0 0) 0x7fa7d8138720 con 0x7fa7d80a98f0 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7c4004eb0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7c8001ae0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.770+0000 7fa7d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.771 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.772 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:31.772 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:31.772 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:31.772 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107913024 0 0) 0x7fa7d8134690 con 0x7fa7d81430c0 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d8138720 con 0x7fa7d81430c0 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7d0001a70 con 0x7fa7d81430c0 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.776+0000 7fa7d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:31.777 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:31.778 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:31.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1857555610 0 0) 0x7fa7c4004eb0 con 0x7fa7d80a98f0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 >> v1:172.21.15.87:6790/0 conn(0x7fa7d813f710 legacy=0x7fa7d8133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 >> v1:172.21.15.87:6789/0 conn(0x7fa7d81430c0 legacy=0x7fa7d81da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 1 monclient: found mon.b 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8134820 con 0x7fa7d80a98f0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 10 monclient: _renew_subs 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:31.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d8138100 con 0x7fa7d80a98f0 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 10 monclient: _renew_subs 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d81daf90 con 0x7fa7d80a98f0 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.778+0000 7fa7de755640 1 librados: init done 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.779+0000 7fa7de755640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.779+0000 7fa7de755640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:31.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.779+0000 7fa7de755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:31.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:31.779+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d8134a10 con 0x7fa7d80a98f0 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c8004a90 con 0x7fa7d80a98f0 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa7c8001c60 con 0x7fa7d80a98f0 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d4ff9640 10 monclient: dump: 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:32.076 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.075+0000 7fa7d67fc640 4 set_mon_vals no callback set 2024-03-25T18:15:32.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.076+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa7c801ac10 con 0x7fa7d80a98f0 2024-03-25T18:15:32.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.076+0000 7fa7d77fe640 10 monclient: get_auth_request con 0x7fa7c406e160 auth_method 0 2024-03-25T18:15:32.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.077+0000 7fa7d4ff9640 10 client.4261.objecter ms_handle_connect 0x7fa7c406e160 2024-03-25T18:15:32.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.080+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fa7c80547c0 con 0x7fa7d80a98f0 2024-03-25T18:15:32.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.080+0000 7fa7d4ff9640 10 client.4261.objecter ms_dispatch 0x7fa7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:32.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.080+0000 7fa7d4ff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:32.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.080+0000 7fa7d4ff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:32.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.080+0000 7fa7d4ff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:15:32.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.081+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7c80502d0 con 0x7fa7d80a98f0 2024-03-25T18:15:32.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.081+0000 7fa7d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:32.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.081+0000 7fa7d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:32.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.308+0000 7fa7de755640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.308+0000 7fa7de755640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.308+0000 7fa7de755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:32.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.308+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7d81388b0 con 0x7fa7d80a98f0 2024-03-25T18:15:32.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.343+0000 7fa7d4ff9640 1 -- 172.21.15.87:0/1452054185 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fa7c8052200 con 0x7fa7d80a98f0 2024-03-25T18:15:32.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.344+0000 7fa7d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:32.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.344+0000 7fa7d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:32.345 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:32.345 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:32.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 10 librados: watch_flush enter 2024-03-25T18:15:32.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 10 librados: watch_flush exit 2024-03-25T18:15:32.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:15:32.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:15:32.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa7c406e160 legacy=0x7fa7c4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 10 monclient: shutdown 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 >> v1:172.21.15.171:6789/0 conn(0x7fa7d80a98f0 legacy=0x7fa7d8137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 shutdown_connections 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 >> 172.21.15.87:0/1452054185 conn(0x7fa7d80b7ea0 msgr2=0x7fa7d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 shutdown_connections 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 -- 172.21.15.87:0/1452054185 wait complete. 2024-03-25T18:15:32.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:32.346+0000 7fa7de755640 1 librados: shutdown 2024-03-25T18:15:32.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:36.285 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:23.999 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.361 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:24.074 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.378 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:24.057 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.433 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:24.145 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.443 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:24.122 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.494 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:24.207 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.596 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:24.309 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.670 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:24.348 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.733 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:24.446 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:39.172 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:39.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.284+0000 7f228ce72640 10 monclient: get_monmap_and_config 2024-03-25T18:15:39.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.284+0000 7f228ce72640 10 monclient: build_initial_monmap 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.284+0000 7f228ce72640 10 monclient: monmap: 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:39.285737+0000 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:39.285737+0000 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:39.286 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.285+0000 7f228ce72640 1 Processor -- start 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.285+0000 7f228ce72640 1 -- start start 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.285+0000 7f228ce72640 10 monclient: init 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-b con 0x7f22880a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-c con 0x7f228813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-a con 0x7f22881430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2288142520 con 0x7f22881430c0 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880b28a0 con 0x7f228813f710 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880b2a30 con 0x7f22880a98f0 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:39.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.286+0000 7f228ce72640 10 monclient(hunting): authenticate will time out at 7854.581759s 2024-03-25T18:15:39.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.296+0000 7f2286d9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f22881430c0 0x7f228814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51664/0 (socket says 172.21.15.87:51664) 2024-03-25T18:15:39.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.297+0000 7f2286d9d640 1 -- 172.21.15.87:0/3469842555 learned_addr learned my addr 172.21.15.87:0/3469842555 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:39.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373701415 0 0) 0x7f2288142520 con 0x7f22881430c0 2024-03-25T18:15:39.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:15:39.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.311+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270003520 con 0x7f22881430c0 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219006329 0 0) 0x7f22880b2a30 con 0x7f22880a98f0 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:15:39.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.316+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288142520 con 0x7f22880a98f0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392612739 0 0) 0x7f2270003520 con 0x7f22881430c0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 >> v1:172.21.15.87:6790/0 conn(0x7f22880a98f0 legacy=0x7f22880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 >> v1:172.21.15.171:6789/0 conn(0x7f228813f710 legacy=0x7f22881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: _finish_hunting 0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 monclient: found mon.noname-a 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880b44c0 con 0x7f22881430c0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: _finish_auth 0 2024-03-25T18:15:39.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: _check_auth_tickets 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2280001a10 con 0x7f22881430c0 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228559a640 10 monclient: dump: 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228ce72640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:15:39.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.318+0000 7f228ce72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:39.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2280001b70 con 0x7f22881430c0 2024-03-25T18:15:39.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:39.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 1 -- 172.21.15.87:0/3469842555 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2280001f80 con 0x7f22881430c0 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228559a640 10 monclient: dump: 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:39.334 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:39.335 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228ce72640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:39.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.333+0000 7f228ce72640 4 set_mon_vals no callback set 2024-03-25T18:15:39.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 10 monclient: shutdown 2024-03-25T18:15:39.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:39.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 1 -- 172.21.15.87:0/3469842555 >> v1:172.21.15.87:6789/0 conn(0x7f22881430c0 legacy=0x7f228814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 1 -- 172.21.15.87:0/3469842555 shutdown_connections 2024-03-25T18:15:39.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 1 -- 172.21.15.87:0/3469842555 >> 172.21.15.87:0/3469842555 conn(0x7f22880b7ea0 msgr2=0x7f22880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 1 -- 172.21.15.87:0/3469842555 shutdown_connections 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.334+0000 7f228ce72640 1 -- 172.21.15.87:0/3469842555 wait complete. 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.335+0000 7f228ce72640 10 monclient: build_initial_monmap 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.335+0000 7f228ce72640 10 monclient: monmap: 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.335+0000 7f228ce72640 1 librados: starting msgr at 2024-03-25T18:15:39.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.335+0000 7f228ce72640 1 librados: starting objecter 2024-03-25T18:15:39.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 Processor -- start 2024-03-25T18:15:39.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 -- start start 2024-03-25T18:15:39.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 librados: setting wanted keys 2024-03-25T18:15:39.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 librados: calling monclient init 2024-03-25T18:15:39.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient: init 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-c con 0x7f22880a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-a con 0x7f228813f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient(hunting): picked mon.noname-b con 0x7f22881430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2288136e20 con 0x7f228813f710 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880b7940 con 0x7f22881430c0 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22881351a0 con 0x7f22880a98f0 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:39.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.336+0000 7f228ce72640 10 monclient(hunting): authenticate will time out at 7854.631617s 2024-03-25T18:15:39.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.344+0000 7f2285d9b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f228813f710 0x7f22881db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:51708/0 (socket says 172.21.15.87:51708) 2024-03-25T18:15:39.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.344+0000 7f2285d9b640 1 -- 172.21.15.87:0/1009309448 learned_addr learned my addr 172.21.15.87:0/1009309448 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:39.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.345+0000 7f2276ffd640 10 client.?.objecter ms_handle_connect 0x7f228813f710 2024-03-25T18:15:39.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.345+0000 7f2276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.352+0000 7f2276ffd640 10 client.?.objecter ms_handle_connect 0x7f22881430c0 2024-03-25T18:15:39.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.352+0000 7f2276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.355+0000 7f2276ffd640 10 client.?.objecter ms_handle_connect 0x7f22880a98f0 2024-03-25T18:15:39.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.355+0000 7f2276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:39.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311829805 0 0) 0x7f22880b7940 con 0x7f22881430c0 2024-03-25T18:15:39.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:15:39.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.358+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2268004eb0 con 0x7f22881430c0 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039717630 0 0) 0x7f2288136e20 con 0x7f228813f710 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:15:39.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.360+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22880b7940 con 0x7f228813f710 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045565176 0 0) 0x7f22881351a0 con 0x7f22880a98f0 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:15:39.365 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.364+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288136e20 con 0x7f22880a98f0 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530559430 0 0) 0x7f2268004eb0 con 0x7f22881430c0 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22881351a0 con 0x7f22881430c0 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2280001ec0 con 0x7f22881430c0 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.371 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.372 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:39.372 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:39.372 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:39.372 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.370+0000 7f2276ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293512865 0 0) 0x7f2288136e20 con 0x7f22880a98f0 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2268004eb0 con 0x7f22880a98f0 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2278001930 con 0x7f22880a98f0 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.381+0000 7f2276ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.384 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3905988995 0 0) 0x7f22881351a0 con 0x7f22881430c0 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 >> v1:172.21.15.87:6790/0 conn(0x7f22880a98f0 legacy=0x7f22881daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 >> v1:172.21.15.87:6789/0 conn(0x7f228813f710 legacy=0x7f22881db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 1 monclient: found mon.b 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:39.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288135330 con 0x7f22881430c0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f2276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f228ce72640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f228ce72640 10 monclient: _renew_subs 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f228ce72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.382+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2288135560 con 0x7f22881430c0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.383+0000 7f228ce72640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.383+0000 7f228ce72640 10 monclient: _renew_subs 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.383+0000 7f228ce72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.383+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f228813b490 con 0x7f22881430c0 2024-03-25T18:15:39.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.383+0000 7f228ce72640 1 librados: init done 2024-03-25T18:15:39.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.384+0000 7f228ce72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.384+0000 7f228ce72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.384+0000 7f228ce72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:39.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.384+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22880b7730 con 0x7f22881430c0 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22800020c0 con 0x7f22881430c0 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f22800015d0 con 0x7f22881430c0 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2276ffd640 10 monclient: dump: 2024-03-25T18:15:39.675 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:39.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.674+0000 7f2284d99640 4 set_mon_vals no callback set 2024-03-25T18:15:39.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.675+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f228001aee0 con 0x7f22881430c0 2024-03-25T18:15:39.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.676+0000 7f228659c640 10 monclient: get_auth_request con 0x7f226806df70 auth_method 0 2024-03-25T18:15:39.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.676+0000 7f2276ffd640 10 client.4267.objecter ms_handle_connect 0x7f226806df70 2024-03-25T18:15:39.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.680+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f2280054b40 con 0x7f22881430c0 2024-03-25T18:15:39.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.680+0000 7f2276ffd640 10 client.4267.objecter ms_dispatch 0x7f2288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:39.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.680+0000 7f2276ffd640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:39.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.680+0000 7f2276ffd640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:39.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.680+0000 7f2276ffd640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T18:15:39.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.682+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2280050480 con 0x7f22881430c0 2024-03-25T18:15:39.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.682+0000 7f2276ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:39.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.682+0000 7f2276ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:39.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.906+0000 7f228ce72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.906+0000 7f228ce72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.906+0000 7f228ce72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:39.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.906+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22880b7ad0 con 0x7f22881430c0 2024-03-25T18:15:39.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.931+0000 7f2276ffd640 1 -- 172.21.15.87:0/1009309448 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f2280052580 con 0x7f22881430c0 2024-03-25T18:15:39.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.931+0000 7f2276ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:39.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.931+0000 7f2276ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:39.933 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:39.933 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:39.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 10 librados: watch_flush enter 2024-03-25T18:15:39.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 10 librados: watch_flush exit 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 10 client.4267.objecter successfully canceled tick 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 >> v1:172.21.15.171:6800/3417775508 conn(0x7f226806df70 legacy=0x7f2268070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 10 monclient: shutdown 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 >> v1:172.21.15.171:6789/0 conn(0x7f22881430c0 legacy=0x7f22881e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 shutdown_connections 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.934+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 >> 172.21.15.87:0/1009309448 conn(0x7f22880b7ea0 msgr2=0x7f2288074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.935+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 shutdown_connections 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.935+0000 7f228ce72640 1 -- 172.21.15.87:0/1009309448 wait complete. 2024-03-25T18:15:39.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:39.935+0000 7f228ce72640 1 librados: shutdown 2024-03-25T18:15:39.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:46.753 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:46.870 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:46.870 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 10 monclient: monmap: 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:46.870951+0000 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:46.870951+0000 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:46.871 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 1 Processor -- start 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 1 -- start start 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.870+0000 7fb69940d640 10 monclient: init 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:46.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6940a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-a con 0x7fb69413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6941430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb694142520 con 0x7fb69413f710 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940b28a0 con 0x7fb6940a98f0 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940b2a30 con 0x7fb6941430c0 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.871+0000 7fb69940d640 10 monclient(hunting): authenticate will time out at 7862.166574s 2024-03-25T18:15:46.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.880+0000 7fb68bfff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb69413f710 0x7fb6941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:56338/0 (socket says 172.21.15.87:56338) 2024-03-25T18:15:46.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.880+0000 7fb68bfff640 1 -- 172.21.15.87:0/596955347 learned_addr learned my addr 172.21.15.87:0/596955347 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553196860 0 0) 0x7fb694142520 con 0x7fb69413f710 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:15:46.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.889+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb680003520 con 0x7fb69413f710 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909892869 0 0) 0x7fb6940b28a0 con 0x7fb6940a98f0 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:15:46.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.890+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb694142520 con 0x7fb6940a98f0 2024-03-25T18:15:46.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219259924 0 0) 0x7fb680003520 con 0x7fb69413f710 2024-03-25T18:15:46.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 >> v1:172.21.15.87:6790/0 conn(0x7fb6941430c0 legacy=0x7fb69414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 >> v1:172.21.15.171:6789/0 conn(0x7fb6940a98f0 legacy=0x7fb6940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940b44c0 con 0x7fb69413f710 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb67c001690 con 0x7fb69413f710 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb6927fc640 10 monclient: dump: 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.901 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb69940d640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:15:46.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.899+0000 7fb69940d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb67c0017f0 con 0x7fb69413f710 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 1 -- 172.21.15.87:0/596955347 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb67c001c00 con 0x7fb69413f710 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb6927fc640 10 monclient: dump: 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:46.904 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb69940d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:46.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.903+0000 7fb69940d640 4 set_mon_vals no callback set 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 10 monclient: shutdown 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 1 -- 172.21.15.87:0/596955347 >> v1:172.21.15.87:6789/0 conn(0x7fb69413f710 legacy=0x7fb6941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 1 -- 172.21.15.87:0/596955347 shutdown_connections 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 1 -- 172.21.15.87:0/596955347 >> 172.21.15.87:0/596955347 conn(0x7fb6940b7ea0 msgr2=0x7fb6940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:46.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 1 -- 172.21.15.87:0/596955347 shutdown_connections 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.904+0000 7fb69940d640 1 -- 172.21.15.87:0/596955347 wait complete. 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 10 monclient: build_initial_monmap 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 10 monclient: monmap: 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 librados: starting msgr at 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 librados: starting objecter 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 Processor -- start 2024-03-25T18:15:46.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 -- start start 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 librados: setting wanted keys 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 1 librados: calling monclient init 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.905+0000 7fb69940d640 10 monclient: init 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6940a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-a con 0x7fb69413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6941430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940abd70 con 0x7fb69413f710 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6941d7710 con 0x7fb6941430c0 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6941d8340 con 0x7fb6940a98f0 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:46.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.906+0000 7fb69940d640 10 monclient(hunting): authenticate will time out at 7862.201248s 2024-03-25T18:15:46.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.915+0000 7fb68bfff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb69413f710 0x7fb6941d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:56374/0 (socket says 172.21.15.87:56374) 2024-03-25T18:15:46.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.915+0000 7fb68bfff640 1 -- 172.21.15.87:0/3184962375 learned_addr learned my addr 172.21.15.87:0/3184962375 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:46.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.918+0000 7fb68b7fe640 10 client.?.objecter ms_handle_connect 0x7fb6941430c0 2024-03-25T18:15:46.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.918+0000 7fb68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.921+0000 7fb68b7fe640 10 client.?.objecter ms_handle_connect 0x7fb69413f710 2024-03-25T18:15:46.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.921+0000 7fb68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835179764 0 0) 0x7fb6940abd70 con 0x7fb69413f710 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:15:46.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.928+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb674004eb0 con 0x7fb69413f710 2024-03-25T18:15:46.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.930+0000 7fb68b7fe640 10 client.?.objecter ms_handle_connect 0x7fb6940a98f0 2024-03-25T18:15:46.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.930+0000 7fb68b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434152794 0 0) 0x7fb6941d7710 con 0x7fb6941430c0 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:15:46.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.934+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6940abd70 con 0x7fb6941430c0 2024-03-25T18:15:46.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398678244 0 0) 0x7fb674004eb0 con 0x7fb69413f710 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6941d7710 con 0x7fb69413f710 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb67c001c40 con 0x7fb69413f710 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:46.943 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:46.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.942+0000 7fb68b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160078776 0 0) 0x7fb6941d8340 con 0x7fb6940a98f0 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:15:46.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.946+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb674004eb0 con 0x7fb6940a98f0 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.954+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863863822 0 0) 0x7fb6941d7710 con 0x7fb69413f710 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 >> v1:172.21.15.87:6790/0 conn(0x7fb6940a98f0 legacy=0x7fb694137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 >> v1:172.21.15.171:6789/0 conn(0x7fb6941430c0 legacy=0x7fb6941d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 1 monclient: found mon.a 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6941d84d0 con 0x7fb69413f710 2024-03-25T18:15:46.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb68b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 5 monclient: authenticate success, global_id 4269 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 10 monclient: _renew_subs 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6941d41e0 con 0x7fb69413f710 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 10 monclient: _renew_subs 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6941d5150 con 0x7fb69413f710 2024-03-25T18:15:46.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.955+0000 7fb69940d640 1 librados: init done 2024-03-25T18:15:46.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.956+0000 7fb69940d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.956+0000 7fb69940d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:46.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.956+0000 7fb69940d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:46.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:46.956+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6940627b0 con 0x7fb69413f710 2024-03-25T18:15:47.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb67c004a90 con 0x7fb69413f710 2024-03-25T18:15:47.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb67c004de0 con 0x7fb69413f710 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb68b7fe640 10 monclient: dump: 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:47.259 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.257+0000 7fb691ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.258+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fb67c005000 con 0x7fb69413f710 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.258+0000 7fb692ffd640 10 monclient: get_auth_request con 0x7fb67406de20 auth_method 0 2024-03-25T18:15:47.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.259+0000 7fb68b7fe640 10 client.4269.objecter ms_handle_connect 0x7fb67406de20 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fb67c054970 con 0x7fb69413f710 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 10 client.4269.objecter ms_dispatch 0x7fb694000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T18:15:47.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb67c050480 con 0x7fb69413f710 2024-03-25T18:15:47.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:47.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.268+0000 7fb68b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:47.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.493+0000 7fb69940d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.493+0000 7fb69940d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.493+0000 7fb69940d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:15:47.495 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.493+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6941d78a0 con 0x7fb69413f710 2024-03-25T18:15:47.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.525+0000 7fb68b7fe640 1 -- 172.21.15.87:0/3184962375 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fb67c0523b0 con 0x7fb69413f710 2024-03-25T18:15:47.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.525+0000 7fb68b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:47.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.525+0000 7fb68b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:47.526 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:47.527 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:47.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 10 librados: watch_flush enter 2024-03-25T18:15:47.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 10 librados: watch_flush exit 2024-03-25T18:15:47.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T18:15:47.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 10 client.4269.objecter successfully canceled tick 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 >> v1:172.21.15.171:6800/3417775508 conn(0x7fb67406de20 legacy=0x7fb6740702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 10 monclient: shutdown 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 >> v1:172.21.15.87:6789/0 conn(0x7fb69413f710 legacy=0x7fb6941d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 shutdown_connections 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 >> 172.21.15.87:0/3184962375 conn(0x7fb6940b7ea0 msgr2=0x7fb694074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 shutdown_connections 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 -- 172.21.15.87:0/3184962375 wait complete. 2024-03-25T18:15:47.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:47.528+0000 7fb69940d640 1 librados: shutdown 2024-03-25T18:15:47.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:54.347 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.462+0000 7fd576f8f640 10 monclient: get_monmap_and_config 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.462+0000 7fd576f8f640 10 monclient: build_initial_monmap 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.462+0000 7fd576f8f640 10 monclient: monmap: 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:15:54.464365+0000 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:15:54.464365+0000 2024-03-25T18:15:54.464 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:54.465 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.465 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:54.465 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:15:54.465 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:15:54.465 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 1 Processor -- start 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 1 -- start start 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 10 monclient: init 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd5700b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.463+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd5700b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:54.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-b con 0x7fd57014be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:54.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700af940 con 0x7fd5700b4d60 2024-03-25T18:15:54.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b0a70 con 0x7fd5700b1730 2024-03-25T18:15:54.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b0d50 con 0x7fd57014be90 2024-03-25T18:15:54.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:54.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.464+0000 7fd576f8f640 10 monclient(hunting): authenticate will time out at 7869.760000s 2024-03-25T18:15:54.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.468+0000 7fd57552c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd57014be90 0x7fd57014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39658/0 (socket says 172.21.15.87:39658) 2024-03-25T18:15:54.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.468+0000 7fd57552c640 1 -- 172.21.15.87:0/4045544463 learned_addr learned my addr 172.21.15.87:0/4045544463 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:54.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549294638 0 0) 0x7fd5700b0d50 con 0x7fd57014be90 2024-03-25T18:15:54.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:15:54.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.479+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd560003520 con 0x7fd57014be90 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908278525 0 0) 0x7fd5700b0a70 con 0x7fd5700b1730 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:15:54.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.484+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5700b0d50 con 0x7fd5700b1730 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012630621 0 0) 0x7fd5700af940 con 0x7fd5700b4d60 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:15:54.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.490+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5700b0a70 con 0x7fd5700b4d60 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686168931 0 0) 0x7fd5700b0d50 con 0x7fd5700b1730 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 >> v1:172.21.15.87:6790/0 conn(0x7fd57014be90 legacy=0x7fd57014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 >> v1:172.21.15.87:6789/0 conn(0x7fd5700b4d60 legacy=0x7fd57013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:15:54.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700b1030 con 0x7fd5700b1730 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd55c001670 con 0x7fd5700b1730 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd576f8f640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd576f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd56f7fe640 10 monclient: dump: 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:54.500 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.501 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.497+0000 7fd576f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:54.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd55c0017d0 con 0x7fd5700b1730 2024-03-25T18:15:54.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:54.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 1 -- 172.21.15.87:0/4045544463 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd55c001be0 con 0x7fd5700b1730 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd576f8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd576f8f640 4 set_mon_vals no callback set 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.515+0000 7fd56f7fe640 10 monclient: dump: 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.518 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.519 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:54.519 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:54.519 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:54.519 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 10 monclient: shutdown 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 1 -- 172.21.15.87:0/4045544463 >> v1:172.21.15.171:6789/0 conn(0x7fd5700b1730 legacy=0x7fd5700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 1 -- 172.21.15.87:0/4045544463 shutdown_connections 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 1 -- 172.21.15.87:0/4045544463 >> 172.21.15.87:0/4045544463 conn(0x7fd5700617a0 msgr2=0x7fd570061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 1 -- 172.21.15.87:0/4045544463 shutdown_connections 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.516+0000 7fd576f8f640 1 -- 172.21.15.87:0/4045544463 wait complete. 2024-03-25T18:15:54.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.517+0000 7fd576f8f640 10 monclient: build_initial_monmap 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.517+0000 7fd576f8f640 10 monclient: monmap: 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 librados: starting msgr at 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 librados: starting objecter 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 Processor -- start 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 -- start start 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 librados: setting wanted keys 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 1 librados: calling monclient init 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.518+0000 7fd576f8f640 10 monclient: init 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-b con 0x7fd5700b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd5700b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd57014be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5701447e0 con 0x7fd5700b4d60 2024-03-25T18:15:54.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd570143900 con 0x7fd5700b1730 2024-03-25T18:15:54.524 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd570143a90 con 0x7fd57014be90 2024-03-25T18:15:54.524 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:54.524 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.519+0000 7fd576f8f640 10 monclient(hunting): authenticate will time out at 7869.815345s 2024-03-25T18:15:54.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.526+0000 7fd57552c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd57014be90 0x7fd5701db2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39698/0 (socket says 172.21.15.87:39698) 2024-03-25T18:15:54.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.526+0000 7fd57552c640 1 -- 172.21.15.87:0/2903792626 learned_addr learned my addr 172.21.15.87:0/2903792626 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:15:54.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.527+0000 7fd56d7fa640 10 client.?.objecter ms_handle_connect 0x7fd57014be90 2024-03-25T18:15:54.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.527+0000 7fd56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.538+0000 7fd56d7fa640 10 client.?.objecter ms_handle_connect 0x7fd5700b1730 2024-03-25T18:15:54.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.538+0000 7fd56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.547 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.546+0000 7fd56d7fa640 10 client.?.objecter ms_handle_connect 0x7fd5700b4d60 2024-03-25T18:15:54.547 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.546+0000 7fd56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385078276 0 0) 0x7fd570143a90 con 0x7fd57014be90 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd554004eb0 con 0x7fd57014be90 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596091646 0 0) 0x7fd570143900 con 0x7fd5700b1730 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:15:54.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.550+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd570143a90 con 0x7fd5700b1730 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2060386333 0 0) 0x7fd570143a90 con 0x7fd5700b1730 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd570143900 con 0x7fd5700b1730 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd564001870 con 0x7fd5700b1730 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.561 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.559+0000 7fd56d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280075375 0 0) 0x7fd5701447e0 con 0x7fd5700b4d60 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:15:54.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.560+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd570143a90 con 0x7fd5700b4d60 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960590333 0 0) 0x7fd554004eb0 con 0x7fd57014be90 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5701447e0 con 0x7fd57014be90 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd568001b30 con 0x7fd57014be90 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.569+0000 7fd56d7fa640 10 monclient(hunting): dump: 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:54.571 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:54.572 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:54.572 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.573 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2980238800 0 0) 0x7fd570143900 con 0x7fd5700b1730 2024-03-25T18:15:54.573 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 >> v1:172.21.15.87:6790/0 conn(0x7fd57014be90 legacy=0x7fd5701db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 >> v1:172.21.15.87:6789/0 conn(0x7fd5700b4d60 legacy=0x7fd5701dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 1 monclient: found mon.b 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570130600 con 0x7fd5700b1730 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd56d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 10 monclient: _renew_subs 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd570144170 con 0x7fd5700b1730 2024-03-25T18:15:54.574 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 10 monclient: _renew_subs 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57013ea40 con 0x7fd5700b1730 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.571+0000 7fd576f8f640 1 librados: init done 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.572+0000 7fd576f8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.572+0000 7fd576f8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.572+0000 7fd576f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:54.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.572+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5700627b0 con 0x7fd5700b1730 2024-03-25T18:15:54.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd564003610 con 0x7fd5700b1730 2024-03-25T18:15:54.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:54.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd564001d80 con 0x7fd5700b1730 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56d7fa640 10 monclient: dump: 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:15:54.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.854+0000 7fd56effd640 4 set_mon_vals no callback set 2024-03-25T18:15:54.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.855+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd564003b20 con 0x7fd5700b1730 2024-03-25T18:15:54.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.856+0000 7fd56ffff640 10 monclient: get_auth_request con 0x7fd55406dfb0 auth_method 0 2024-03-25T18:15:54.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.856+0000 7fd56d7fa640 10 client.4279.objecter ms_handle_connect 0x7fd55406dfb0 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd564053540 con 0x7fd5700b1730 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 10 client.4279.objecter ms_dispatch 0x7fd570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd56404f050 con 0x7fd5700b1730 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:54.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:54.865+0000 7fd56d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:55.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.117+0000 7fd576f8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:55.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.117+0000 7fd576f8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:55.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.117+0000 7fd576f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:15:55.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.117+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd57013ef10 con 0x7fd5700b1730 2024-03-25T18:15:55.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.143+0000 7fd56d7fa640 1 -- 172.21.15.87:0/2903792626 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd564056ba0 con 0x7fd5700b1730 2024-03-25T18:15:55.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.143+0000 7fd56d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:55.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.143+0000 7fd56d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:55.146 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:15:55.146 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.145+0000 7fd576f8f640 10 librados: watch_flush enter 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 10 librados: watch_flush exit 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 10 client.4279.objecter successfully canceled tick 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd55406dfb0 legacy=0x7fd554070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 10 monclient: shutdown 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 >> v1:172.21.15.171:6789/0 conn(0x7fd5700b1730 legacy=0x7fd57012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:55.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 shutdown_connections 2024-03-25T18:15:55.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 >> 172.21.15.87:0/2903792626 conn(0x7fd5700617a0 msgr2=0x7fd5700b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:55.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 shutdown_connections 2024-03-25T18:15:55.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 -- 172.21.15.87:0/2903792626 wait complete. 2024-03-25T18:15:55.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:15:55.146+0000 7fd576f8f640 1 librados: shutdown 2024-03-25T18:15:55.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:56.322 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:44.035 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.329 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:44.042 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.387 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:01:44.100 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.397 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:44.076 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.405 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:44.084 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.405 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:44.119 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.408 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:44.121 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.457 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:01:44.136 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.465 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:01:44.179 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:01.968 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:02.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.083+0000 7fe3aad38640 10 monclient: get_monmap_and_config 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.083+0000 7fe3aad38640 10 monclient: build_initial_monmap 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.083+0000 7fe3aad38640 10 monclient: monmap: 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:02.084784+0000 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:02.084784+0000 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:02.085 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:02.086 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.084+0000 7fe3aad38640 1 Processor -- start 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 1 -- start start 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient: init 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a40705b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a41402d0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a4149ae0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40718c0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40714c0 con 0x7fe3a41402d0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a4072b50 con 0x7fe3a40705b0 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:02.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.085+0000 7fe3aad38640 10 monclient(hunting): authenticate will time out at 7877.380653s 2024-03-25T18:16:02.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.089+0000 7fe3a8ad4640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fe3a40705b0 0x7fe3a413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:47952/0 (socket says 172.21.15.87:47952) 2024-03-25T18:16:02.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.089+0000 7fe3a8ad4640 1 -- 172.21.15.87:0/1506990873 learned_addr learned my addr 172.21.15.87:0/1506990873 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154268874 0 0) 0x7fe3a4072b50 con 0x7fe3a40705b0 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:16:02.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.102+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe394003520 con 0x7fe3a40705b0 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244122954 0 0) 0x7fe3a40714c0 con 0x7fe3a41402d0 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:16:02.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.113+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4072b50 con 0x7fe3a41402d0 2024-03-25T18:16:02.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906528335 0 0) 0x7fe394003520 con 0x7fe3a40705b0 2024-03-25T18:16:02.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 >> v1:172.21.15.171:6789/0 conn(0x7fe3a41402d0 legacy=0x7fe3a4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 >> v1:172.21.15.87:6789/0 conn(0x7fe3a4149ae0 legacy=0x7fe3a414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 monclient: found mon.noname-b 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a413dd10 con 0x7fe3a40705b0 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe398001690 con 0x7fe3a40705b0 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3a37fe640 10 monclient: dump: 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:02.116 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3aad38640 5 monclient: authenticate success, global_id 4283 2024-03-25T18:16:02.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.114+0000 7fe3aad38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:02.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3980017f0 con 0x7fe3a40705b0 2024-03-25T18:16:02.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 1 -- 172.21.15.87:0/1506990873 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe398001c00 con 0x7fe3a40705b0 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3aad38640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3aad38640 4 set_mon_vals no callback set 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.133+0000 7fe3a37fe640 10 monclient: dump: 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:02.135 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:02.136 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:02.136 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 10 monclient: shutdown 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 1 -- 172.21.15.87:0/1506990873 >> v1:172.21.15.87:6790/0 conn(0x7fe3a40705b0 legacy=0x7fe3a413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 1 -- 172.21.15.87:0/1506990873 shutdown_connections 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 1 -- 172.21.15.87:0/1506990873 >> 172.21.15.87:0/1506990873 conn(0x7fe3a40b5d10 msgr2=0x7fe3a40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 1 -- 172.21.15.87:0/1506990873 shutdown_connections 2024-03-25T18:16:02.138 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.134+0000 7fe3aad38640 1 -- 172.21.15.87:0/1506990873 wait complete. 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 10 monclient: build_initial_monmap 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 10 monclient: monmap: 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:02.139 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 librados: starting msgr at 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 librados: starting objecter 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 Processor -- start 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 -- start start 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 librados: setting wanted keys 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 1 librados: calling monclient init 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.135+0000 7fe3aad38640 10 monclient: init 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a40705b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:02.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a41402d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a4149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a413e460 con 0x7fe3a40705b0 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a413dfd0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a413e160 con 0x7fe3a41402d0 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:02.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.136+0000 7fe3aad38640 10 monclient(hunting): authenticate will time out at 7877.431615s 2024-03-25T18:16:02.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.142+0000 7fe3a3fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fe3a41402d0 0x7fe3a406d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:47984/0 (socket says 172.21.15.87:47984) 2024-03-25T18:16:02.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.142+0000 7fe3a3fff640 1 -- 172.21.15.87:0/1726704772 learned_addr learned my addr 172.21.15.87:0/1726704772 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:02.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.144+0000 7fe3a17fa640 10 client.?.objecter ms_handle_connect 0x7fe3a41402d0 2024-03-25T18:16:02.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.144+0000 7fe3a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.146+0000 7fe3a17fa640 10 client.?.objecter ms_handle_connect 0x7fe3a4149ae0 2024-03-25T18:16:02.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.146+0000 7fe3a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.147+0000 7fe3a17fa640 10 client.?.objecter ms_handle_connect 0x7fe3a40705b0 2024-03-25T18:16:02.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.147+0000 7fe3a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480073458 0 0) 0x7fe3a413dfd0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:16:02.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.155+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe388004eb0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.159 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490707486 0 0) 0x7fe3a413e160 con 0x7fe3a41402d0 2024-03-25T18:16:02.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:16:02.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.159+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a413dfd0 con 0x7fe3a41402d0 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452129279 0 0) 0x7fe3a413e460 con 0x7fe3a40705b0 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:16:02.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.161+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a413e160 con 0x7fe3a40705b0 2024-03-25T18:16:02.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3779608448 0 0) 0x7fe388004eb0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3a413e460 con 0x7fe3a4149ae0 2024-03-25T18:16:02.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe39c001b90 con 0x7fe3a4149ae0 2024-03-25T18:16:02.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 10 monclient(hunting): dump: 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.162+0000 7fe3a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:02.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642133819 0 0) 0x7fe3a413e460 con 0x7fe3a4149ae0 2024-03-25T18:16:02.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 >> v1:172.21.15.87:6790/0 conn(0x7fe3a41402d0 legacy=0x7fe3a406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 >> v1:172.21.15.87:6789/0 conn(0x7fe3a40705b0 legacy=0x7fe3a4134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 1 monclient: found mon.b 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4148ee0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 10 monclient: _renew_subs 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:02.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a4149150 con 0x7fe3a4149ae0 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 10 monclient: _renew_subs 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a406db10 con 0x7fe3a4149ae0 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.174+0000 7fe3aad38640 1 librados: init done 2024-03-25T18:16:02.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.175+0000 7fe3aad38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.175+0000 7fe3aad38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.175+0000 7fe3aad38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:02.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.176+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a413e5f0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe39c002080 con 0x7fe3a4149ae0 2024-03-25T18:16:02.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:02.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe39c003b30 con 0x7fe3a4149ae0 2024-03-25T18:16:02.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a17fa640 10 monclient: dump: 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:02.199 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:02.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.197+0000 7fe3a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:02.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.198+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fe39c003d50 con 0x7fe3a4149ae0 2024-03-25T18:16:02.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.199+0000 7fe3a8ad4640 10 monclient: get_auth_request con 0x7fe38806ddf0 auth_method 0 2024-03-25T18:16:02.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.199+0000 7fe3a17fa640 10 client.4285.objecter ms_handle_connect 0x7fe38806ddf0 2024-03-25T18:16:02.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.201+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fe39c053810 con 0x7fe3a4149ae0 2024-03-25T18:16:02.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.201+0000 7fe3a17fa640 10 client.4285.objecter ms_dispatch 0x7fe3a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:02.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.201+0000 7fe3a17fa640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:02.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.201+0000 7fe3a17fa640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:02.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.201+0000 7fe3a17fa640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T18:16:02.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.460+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe39c04f320 con 0x7fe3a4149ae0 2024-03-25T18:16:02.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.460+0000 7fe3a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:02.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.460+0000 7fe3a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:02.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.708+0000 7fe3aad38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.708+0000 7fe3aad38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.708+0000 7fe3aad38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:02.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.708+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3a40627b0 con 0x7fe3a4149ae0 2024-03-25T18:16:02.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.732+0000 7fe3a17fa640 1 -- 172.21.15.87:0/1726704772 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fe39c051250 con 0x7fe3a4149ae0 2024-03-25T18:16:02.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.732+0000 7fe3a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:02.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.732+0000 7fe3a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:02.733 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:02.733 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 10 librados: watch_flush enter 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 10 librados: watch_flush exit 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 10 client.4285.objecter successfully canceled tick 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 >> v1:172.21.15.171:6800/3417775508 conn(0x7fe38806ddf0 legacy=0x7fe3880702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 10 monclient: shutdown 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.734+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 >> v1:172.21.15.171:6789/0 conn(0x7fe3a4149ae0 legacy=0x7fe3a41d7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:02.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.735+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 shutdown_connections 2024-03-25T18:16:02.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.735+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 >> 172.21.15.87:0/1726704772 conn(0x7fe3a40b5d10 msgr2=0x7fe3a413c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:02.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.735+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 shutdown_connections 2024-03-25T18:16:02.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.735+0000 7fe3aad38640 1 -- 172.21.15.87:0/1726704772 wait complete. 2024-03-25T18:16:02.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:02.735+0000 7fe3aad38640 1 librados: shutdown 2024-03-25T18:16:02.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:09.552 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:09.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.667+0000 7f4fcb6a8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.667+0000 7f4fcb6a8640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.667+0000 7f4fcb6a8640 10 monclient: monmap: 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:09.668018+0000 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:09.668018+0000 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:09.668 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.669 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.668+0000 7f4fcb6a8640 1 Processor -- start 2024-03-25T18:16:09.669 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.668+0000 7f4fcb6a8640 1 -- start start 2024-03-25T18:16:09.669 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.668+0000 7f4fcb6a8640 10 monclient: init 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.668+0000 7f4fcb6a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.668+0000 7f4fcb6a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4fc40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4fc413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4fc41430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc4142520 con 0x7f4fc413f710 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40b28a0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40b2a30 con 0x7f4fc41430c0 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.669+0000 7f4fcb6a8640 10 monclient(hunting): authenticate will time out at 7884.963944s 2024-03-25T18:16:09.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.677+0000 7f4fc9444640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4fc40a98f0 0x7f4fc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:40276/0 (socket says 172.21.15.87:40276) 2024-03-25T18:16:09.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.677+0000 7f4fc9444640 1 -- 172.21.15.87:0/4271436071 learned_addr learned my addr 172.21.15.87:0/4271436071 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735666177 0 0) 0x7f4fc40b28a0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:16:09.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.684+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb0003520 con 0x7f4fc40a98f0 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491117209 0 0) 0x7f4fb0003520 con 0x7f4fc40a98f0 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 >> v1:172.21.15.87:6790/0 conn(0x7f4fc41430c0 legacy=0x7f4fc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 >> v1:172.21.15.87:6789/0 conn(0x7f4fc413f710 legacy=0x7f4fc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 monclient: found mon.noname-c 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc40b44c0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fc0001670 con 0x7f4fc40a98f0 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fcb6a8640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fcb6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fbbfff640 10 monclient: dump: 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.693 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:09.694 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:09.694 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:09.694 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.691+0000 7f4fcb6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fc00017d0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 1 -- 172.21.15.87:0/4271436071 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fc0001c20 con 0x7f4fc40a98f0 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fcb6a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fcb6a8640 4 set_mon_vals no callback set 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.706+0000 7f4fbbfff640 10 monclient: dump: 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:09.708 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 10 monclient: shutdown 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/4271436071 >> v1:172.21.15.171:6789/0 conn(0x7f4fc40a98f0 legacy=0x7f4fc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/4271436071 shutdown_connections 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/4271436071 >> 172.21.15.87:0/4271436071 conn(0x7f4fc40b7ea0 msgr2=0x7f4fc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/4271436071 shutdown_connections 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.707+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/4271436071 wait complete. 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.708+0000 7f4fcb6a8640 10 monclient: build_initial_monmap 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.708+0000 7f4fcb6a8640 10 monclient: monmap: 2024-03-25T18:16:09.711 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 librados: starting msgr at 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 librados: starting objecter 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 Processor -- start 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 -- start start 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 librados: setting wanted keys 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 1 librados: calling monclient init 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 10 monclient: init 2024-03-25T18:16:09.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.709+0000 7f4fcb6a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4fc40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4fc413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4fc41430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc4138b20 con 0x7f4fc413f710 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40aba90 con 0x7f4fc41430c0 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc41d4310 con 0x7f4fc40a98f0 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.710+0000 7f4fcb6a8640 10 monclient(hunting): authenticate will time out at 7885.004905s 2024-03-25T18:16:09.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.719+0000 7f4fc8c43640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4fc413f710 0x7f4fc41d4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45508/0 (socket says 172.21.15.87:45508) 2024-03-25T18:16:09.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.719+0000 7f4fc8c43640 1 -- 172.21.15.87:0/1328431766 learned_addr learned my addr 172.21.15.87:0/1328431766 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:09.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.723+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc413f710 2024-03-25T18:16:09.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.723+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.730+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc41430c0 2024-03-25T18:16:09.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.730+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.731+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc40a98f0 2024-03-25T18:16:09.732 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.731+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536654296 0 0) 0x7f4fc4138b20 con 0x7f4fc413f710 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:16:09.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.736+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8004eb0 con 0x7f4fc413f710 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062025379 0 0) 0x7f4fc40aba90 con 0x7f4fc41430c0 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:16:09.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.741+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc4138b20 con 0x7f4fc41430c0 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991179121 0 0) 0x7f4fc41d4310 con 0x7f4fc40a98f0 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:16:09.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.745+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc40aba90 con 0x7f4fc40a98f0 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2319537954 0 0) 0x7f4fc4138b20 con 0x7f4fc41430c0 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fc41d4310 con 0x7f4fc41430c0 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fbc001b10 con 0x7f4fc41430c0 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:09.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.759+0000 7f4fb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092224093 0 0) 0x7f4fa8004eb0 con 0x7f4fc413f710 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fc4138b20 con 0x7f4fc413f710 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fb40019d0 con 0x7f4fc413f710 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.760+0000 7f4fb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:09.762 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:09.763 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 957548656 0 0) 0x7f4fc40aba90 con 0x7f4fc40a98f0 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fa8004eb0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fc0001ae0 con 0x7f4fc40a98f0 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.767+0000 7f4fb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.768 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:09.769 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003689266 0 0) 0x7f4fc4138b20 con 0x7f4fc413f710 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 >> v1:172.21.15.87:6790/0 conn(0x7f4fc40a98f0 legacy=0x7f4fc4138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 >> v1:172.21.15.171:6789/0 conn(0x7f4fc41430c0 legacy=0x7f4fc4133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 1 monclient: found mon.a 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:16:09.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4135160 con 0x7f4fc413f710 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 10 monclient: _renew_subs 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc4138d90 con 0x7f4fc413f710 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 10 monclient: _renew_subs 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc41d5120 con 0x7f4fc413f710 2024-03-25T18:16:09.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.773+0000 7f4fcb6a8640 1 librados: init done 2024-03-25T18:16:09.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.774+0000 7f4fcb6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.774+0000 7f4fcb6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:09.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.774+0000 7f4fcb6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:16:09.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:09.774+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc40abfb0 con 0x7f4fc413f710 2024-03-25T18:16:10.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fb4003610 con 0x7f4fc413f710 2024-03-25T18:16:10.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:10.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4fb4003840 con 0x7f4fc413f710 2024-03-25T18:16:10.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:10.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fb9ffb640 10 monclient: dump: 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:10.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.077+0000 7f4fbb7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:10.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.078+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4fb4003a60 con 0x7f4fc413f710 2024-03-25T18:16:10.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.079+0000 7f4fc9444640 10 monclient: get_auth_request con 0x7f4fa806e160 auth_method 0 2024-03-25T18:16:10.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.079+0000 7f4fb9ffb640 10 client.4287.objecter ms_handle_connect 0x7f4fa806e160 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4fb4054640 con 0x7f4fc413f710 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 10 client.4287.objecter ms_dispatch 0x7f4fc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fb4050150 con 0x7f4fc413f710 2024-03-25T18:16:10.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:10.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.086+0000 7f4fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:10.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.330+0000 7f4fcb6a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.330+0000 7f4fcb6a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.330+0000 7f4fcb6a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:16:10.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.330+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc4074cc0 con 0x7f4fc413f710 2024-03-25T18:16:10.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.411+0000 7f4fb9ffb640 1 -- 172.21.15.87:0/1328431766 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4fb4052080 con 0x7f4fc413f710 2024-03-25T18:16:10.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.411+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:10.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.411+0000 7f4fb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:10.412 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:10.413 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 10 librados: watch_flush enter 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 10 librados: watch_flush exit 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 10 client.4287.objecter successfully canceled tick 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4fa806e160 legacy=0x7f4fa8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 10 monclient: shutdown 2024-03-25T18:16:10.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 >> v1:172.21.15.87:6789/0 conn(0x7f4fc413f710 legacy=0x7f4fc41d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 shutdown_connections 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 >> 172.21.15.87:0/1328431766 conn(0x7f4fc40b7ea0 msgr2=0x7f4fc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 shutdown_connections 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 -- 172.21.15.87:0/1328431766 wait complete. 2024-03-25T18:16:10.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:10.414+0000 7f4fcb6a8640 1 librados: shutdown 2024-03-25T18:16:10.425 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:16.347 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:04.061 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.401 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:04.114 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.423 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:04.137 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.452 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:04.131 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.457 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:04.136 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.469 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:04.183 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.475 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:04.189 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.482 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:04.161 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:16.511 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:04.224 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.231 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:17.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.349+0000 7f886b874640 10 monclient: get_monmap_and_config 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.349+0000 7f886b874640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.349+0000 7f886b874640 10 monclient: monmap: 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:17.350150+0000 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:17.350150+0000 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:17.350 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.350+0000 7f886b874640 1 Processor -- start 2024-03-25T18:16:17.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.350+0000 7f886b874640 1 -- start start 2024-03-25T18:16:17.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.350+0000 7f886b874640 10 monclient: init 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient(hunting): picked mon.noname-a con 0x7f88640a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient(hunting): picked mon.noname-c con 0x7f886413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient(hunting): picked mon.noname-b con 0x7f88641430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8864142520 con 0x7f88640a98f0 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640b28a0 con 0x7f886413f710 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640b2a30 con 0x7f88641430c0 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.351+0000 7f886b874640 10 monclient(hunting): authenticate will time out at 7892.646334s 2024-03-25T18:16:17.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.356+0000 7f8869e11640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f88641430c0 0x7f886414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:33760/0 (socket says 172.21.15.87:33760) 2024-03-25T18:16:17.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.356+0000 7f8869e11640 1 -- 172.21.15.87:0/1788803246 learned_addr learned my addr 172.21.15.87:0/1788803246 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625747569 0 0) 0x7f88640b2a30 con 0x7f88641430c0 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:16:17.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.362+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8850003520 con 0x7f88641430c0 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176402736 0 0) 0x7f8864142520 con 0x7f88640a98f0 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:16:17.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.373+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88640b2a30 con 0x7f88640a98f0 2024-03-25T18:16:17.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811941813 0 0) 0x7f8850003520 con 0x7f88641430c0 2024-03-25T18:16:17.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 >> v1:172.21.15.171:6789/0 conn(0x7f886413f710 legacy=0x7f88641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 >> v1:172.21.15.87:6789/0 conn(0x7f88640a98f0 legacy=0x7f88640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 monclient: found mon.noname-b 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640b44c0 con 0x7f88641430c0 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f885c001a10 con 0x7f88641430c0 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f885bfff640 10 monclient: dump: 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.378 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f886b874640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:16:17.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.376+0000 7f886b874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f885c001b70 con 0x7f88641430c0 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 1 -- 172.21.15.87:0/1788803246 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f885c001f80 con 0x7f88641430c0 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f886b874640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f886b874640 4 set_mon_vals no callback set 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.392+0000 7f885bfff640 10 monclient: dump: 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.394 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 10 monclient: shutdown 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 1 -- 172.21.15.87:0/1788803246 >> v1:172.21.15.87:6790/0 conn(0x7f88641430c0 legacy=0x7f886414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 1 -- 172.21.15.87:0/1788803246 shutdown_connections 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 1 -- 172.21.15.87:0/1788803246 >> 172.21.15.87:0/1788803246 conn(0x7f88640b7ea0 msgr2=0x7f88640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 1 -- 172.21.15.87:0/1788803246 shutdown_connections 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.393+0000 7f886b874640 1 -- 172.21.15.87:0/1788803246 wait complete. 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.394+0000 7f886b874640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.394+0000 7f886b874640 10 monclient: monmap: 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:17.396 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.394+0000 7f886b874640 1 librados: starting msgr at 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 1 librados: starting objecter 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 1 Processor -- start 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 1 -- start start 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 1 librados: setting wanted keys 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 1 librados: calling monclient init 2024-03-25T18:16:17.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 10 monclient: init 2024-03-25T18:16:17.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.395+0000 7f886b874640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:17.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient(hunting): picked mon.noname-c con 0x7f88640a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient(hunting): picked mon.noname-b con 0x7f886413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient(hunting): picked mon.noname-a con 0x7f88641430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886413c2b0 con 0x7f88641430c0 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8864136f10 con 0x7f886413f710 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88641370a0 con 0x7f88640a98f0 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.396+0000 7f886b874640 10 monclient(hunting): authenticate will time out at 7892.691073s 2024-03-25T18:16:17.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.404+0000 7f8868e0f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f886413f710 0x7f88641377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:44928/0 (socket says 172.21.15.87:44928) 2024-03-25T18:16:17.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.404+0000 7f8868e0f640 1 -- 172.21.15.87:0/682188060 learned_addr learned my addr 172.21.15.87:0/682188060 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:17.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.404+0000 7f8859ffb640 10 client.?.objecter ms_handle_connect 0x7f886413f710 2024-03-25T18:16:17.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.404+0000 7f8859ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.407+0000 7f8859ffb640 10 client.?.objecter ms_handle_connect 0x7f88641430c0 2024-03-25T18:16:17.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.407+0000 7f8859ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.408+0000 7f8859ffb640 10 client.?.objecter ms_handle_connect 0x7f88640a98f0 2024-03-25T18:16:17.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.408+0000 7f8859ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884661868 0 0) 0x7f88641370a0 con 0x7f88640a98f0 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:16:17.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.419+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8848004eb0 con 0x7f88640a98f0 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553285981 0 0) 0x7f8864136f10 con 0x7f886413f710 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:16:17.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.422+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88641370a0 con 0x7f886413f710 2024-03-25T18:16:17.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613655731 0 0) 0x7f886413c2b0 con 0x7f88641430c0 2024-03-25T18:16:17.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:16:17.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.425+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8864136f10 con 0x7f88641430c0 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907575908 0 0) 0x7f88641370a0 con 0x7f886413f710 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f886413c2b0 con 0x7f886413f710 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8854001970 con 0x7f886413f710 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.435 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.436 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.436 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.434+0000 7f8859ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2609441739 0 0) 0x7f8848004eb0 con 0x7f88640a98f0 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88641370a0 con 0x7f88640a98f0 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8860001970 con 0x7f88640a98f0 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.443+0000 7f8859ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.445 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171474729 0 0) 0x7f8864136f10 con 0x7f88641430c0 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8848004eb0 con 0x7f88641430c0 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f885c001e80 con 0x7f88641430c0 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.451+0000 7f8859ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.452 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3120982474 0 0) 0x7f88641370a0 con 0x7f88640a98f0 2024-03-25T18:16:17.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 >> v1:172.21.15.171:6789/0 conn(0x7f886413f710 legacy=0x7f88641377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 >> v1:172.21.15.87:6789/0 conn(0x7f88641430c0 legacy=0x7f8864133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 1 monclient: found mon.c 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864137230 con 0x7f88640a98f0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f8859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f886b874640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f886b874640 10 monclient: _renew_subs 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f886b874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.454+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886413b620 con 0x7f88640a98f0 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.455+0000 7f886b874640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.455+0000 7f886b874640 10 monclient: _renew_subs 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.455+0000 7f886b874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.455+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8864133710 con 0x7f88640a98f0 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.455+0000 7f886b874640 1 librados: init done 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.456+0000 7f886b874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.456+0000 7f886b874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.456+0000 7f886b874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:17.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.456+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88640627b0 con 0x7f88640a98f0 2024-03-25T18:16:17.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8860003610 con 0x7f88640a98f0 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8860003780 con 0x7f88640a98f0 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f8859ffb640 10 monclient: dump: 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:17.780 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:17.781 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:17.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.778+0000 7f885b7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:17.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.779+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f88600197d0 con 0x7f88640a98f0 2024-03-25T18:16:17.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.780+0000 7f8868e0f640 10 monclient: get_auth_request con 0x7f884806e110 auth_method 0 2024-03-25T18:16:17.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.780+0000 7f8859ffb640 10 client.4298.objecter ms_handle_connect 0x7f884806e110 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f88600534b0 con 0x7f88640a98f0 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 10 client.4298.objecter ms_dispatch 0x7f8864000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f886004edf0 con 0x7f88640a98f0 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:17.785+0000 7f8859ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:18.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.039+0000 7f886b874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:18.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.039+0000 7f886b874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:18.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.039+0000 7f886b874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:18.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.039+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88641373f0 con 0x7f88640a98f0 2024-03-25T18:16:18.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.140+0000 7f8859ffb640 1 -- 172.21.15.87:0/682188060 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f8860050ef0 con 0x7f88640a98f0 2024-03-25T18:16:18.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.140+0000 7f8859ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:18.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.140+0000 7f8859ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:18.141 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:18.141 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 10 librados: watch_flush enter 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 10 librados: watch_flush exit 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 10 client.4298.objecter successfully canceled tick 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 >> v1:172.21.15.171:6800/3417775508 conn(0x7f884806e110 legacy=0x7f88480705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 10 monclient: shutdown 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.142+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 >> v1:172.21.15.87:6790/0 conn(0x7f88640a98f0 legacy=0x7f88641d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.143+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 shutdown_connections 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.143+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 >> 172.21.15.87:0/682188060 conn(0x7f88640b7ea0 msgr2=0x7f8864074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.143+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 shutdown_connections 2024-03-25T18:16:18.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.143+0000 7f886b874640 1 -- 172.21.15.87:0/682188060 wait complete. 2024-03-25T18:16:18.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:18.143+0000 7f886b874640 1 librados: shutdown 2024-03-25T18:16:18.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:24.960 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:25.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.072+0000 7fcf21fe6640 10 monclient: get_monmap_and_config 2024-03-25T18:16:25.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.072+0000 7fcf21fe6640 10 monclient: build_initial_monmap 2024-03-25T18:16:25.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.072+0000 7fcf21fe6640 10 monclient: monmap: 2024-03-25T18:16:25.073 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:25.073 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:25.073537+0000 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:25.073537+0000 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:25.074 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.073+0000 7fcf21fe6640 1 Processor -- start 2024-03-25T18:16:25.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.073+0000 7fcf21fe6640 1 -- start start 2024-03-25T18:16:25.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.073+0000 7fcf21fe6640 10 monclient: init 2024-03-25T18:16:25.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:25.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c14be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0af940 con 0x7fcf1c0b1730 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0b0a70 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0b0d50 con 0x7fcf1c14be90 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:25.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.074+0000 7fcf21fe6640 10 monclient(hunting): authenticate will time out at 7900.369564s 2024-03-25T18:16:25.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.092+0000 7fcf1affd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fcf1c0b1730 0x7fcf1c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54966/0 (socket says 172.21.15.87:54966) 2024-03-25T18:16:25.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.092+0000 7fcf1affd640 1 -- 172.21.15.87:0/3231151598 learned_addr learned my addr 172.21.15.87:0/3231151598 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6865013 0 0) 0x7fcf1c0b0a70 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:16:25.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.103+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04003520 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717364365 0 0) 0x7fcf1c0af940 con 0x7fcf1c0b1730 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:16:25.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.109+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c0b0a70 con 0x7fcf1c0b1730 2024-03-25T18:16:25.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408420844 0 0) 0x7fcf1c0b0d50 con 0x7fcf1c14be90 2024-03-25T18:16:25.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:16:25.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.112+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c0af940 con 0x7fcf1c14be90 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33591305 0 0) 0x7fcf04003520 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 >> v1:172.21.15.87:6790/0 conn(0x7fcf1c14be90 legacy=0x7fcf1c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 >> v1:172.21.15.87:6789/0 conn(0x7fcf1c0b1730 legacy=0x7fcf1c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0b1030 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:25.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf0c001690 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf21fe6640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf21fe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf1a7fc640 10 monclient: dump: 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.113+0000 7fcf21fe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf0c0017f0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 1 -- 172.21.15.87:0/3231151598 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf0c001c00 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf1a7fc640 10 monclient: dump: 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.122 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf21fe6640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:25.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.121+0000 7fcf21fe6640 4 set_mon_vals no callback set 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 10 monclient: shutdown 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 1 -- 172.21.15.87:0/3231151598 >> v1:172.21.15.171:6789/0 conn(0x7fcf1c0b4d60 legacy=0x7fcf1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 1 -- 172.21.15.87:0/3231151598 shutdown_connections 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 1 -- 172.21.15.87:0/3231151598 >> 172.21.15.87:0/3231151598 conn(0x7fcf1c0617a0 msgr2=0x7fcf1c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 1 -- 172.21.15.87:0/3231151598 shutdown_connections 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.122+0000 7fcf21fe6640 1 -- 172.21.15.87:0/3231151598 wait complete. 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.123+0000 7fcf21fe6640 10 monclient: build_initial_monmap 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.123+0000 7fcf21fe6640 10 monclient: monmap: 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:25.125 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 librados: starting msgr at 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 librados: starting objecter 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 Processor -- start 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 -- start start 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 librados: setting wanted keys 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 1 librados: calling monclient init 2024-03-25T18:16:25.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.124+0000 7fcf21fe6640 10 monclient: init 2024-03-25T18:16:25.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c142050 con 0x7fcf1c14be90 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c1dac50 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c140d00 con 0x7fcf1c0b1730 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:25.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.125+0000 7fcf21fe6640 10 monclient(hunting): authenticate will time out at 7900.420604s 2024-03-25T18:16:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.137+0000 7fcf1b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fcf1c0b1730 0x7fcf1c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:36080/0 (socket says 172.21.15.87:36080) 2024-03-25T18:16:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.137+0000 7fcf1b7fe640 1 -- 172.21.15.87:0/4248902108 learned_addr learned my addr 172.21.15.87:0/4248902108 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:25.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.139+0000 7fcf03fff640 10 client.?.objecter ms_handle_connect 0x7fcf1c0b1730 2024-03-25T18:16:25.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.139+0000 7fcf03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.139+0000 7fcf03fff640 10 client.?.objecter ms_handle_connect 0x7fcf1c0b4d60 2024-03-25T18:16:25.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.139+0000 7fcf03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.142+0000 7fcf03fff640 10 client.?.objecter ms_handle_connect 0x7fcf1c14be90 2024-03-25T18:16:25.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.142+0000 7fcf03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:25.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968604180 0 0) 0x7fcf1c1dac50 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:16:25.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.154+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf08004eb0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990867473 0 0) 0x7fcf1c140d00 con 0x7fcf1c0b1730 2024-03-25T18:16:25.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.156+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c1dac50 con 0x7fcf1c0b1730 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790083734 0 0) 0x7fcf1c142050 con 0x7fcf1c14be90 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:16:25.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.157+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c140d00 con 0x7fcf1c14be90 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379890237 0 0) 0x7fcf1c1dac50 con 0x7fcf1c0b1730 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf1c142050 con 0x7fcf1c0b1730 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf0c001b40 con 0x7fcf1c0b1730 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 10 monclient(hunting): dump: 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.174 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.171+0000 7fcf03fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190792185 0 0) 0x7fcf08004eb0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf1c1dac50 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf10001930 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.172+0000 7fcf03fff640 10 monclient(hunting): dump: 2024-03-25T18:16:25.175 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.176 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399939458 0 0) 0x7fcf1c140d00 con 0x7fcf1c14be90 2024-03-25T18:16:25.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf08004eb0 con 0x7fcf1c14be90 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf14001bd0 con 0x7fcf1c14be90 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.178+0000 7fcf03fff640 10 monclient(hunting): dump: 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.179 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3917901848 0 0) 0x7fcf1c1dac50 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 >> v1:172.21.15.87:6790/0 conn(0x7fcf1c0b1730 legacy=0x7fcf1c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 >> v1:172.21.15.87:6789/0 conn(0x7fcf1c14be90 legacy=0x7fcf1c1e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 1 monclient: found mon.b 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c1423e0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 10 monclient: _renew_subs 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c13b470 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 10 monclient: _renew_subs 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c13b720 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.179+0000 7fcf21fe6640 1 librados: init done 2024-03-25T18:16:25.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.181+0000 7fcf21fe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.181+0000 7fcf21fe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.181+0000 7fcf21fe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.181+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c1401f0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf100047e0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:25.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fcf10004af0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf03fff640 10 monclient: dump: 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:25.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.458+0000 7fcf19ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:25.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.459+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fcf1001aac0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.460+0000 7fcf1b7fe640 10 monclient: get_auth_request con 0x7fcf0806e160 auth_method 0 2024-03-25T18:16:25.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.460+0000 7fcf03fff640 10 client.4303.objecter ms_handle_connect 0x7fcf0806e160 2024-03-25T18:16:25.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.464+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fcf10054780 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.464+0000 7fcf03fff640 10 client.4303.objecter ms_dispatch 0x7fcf1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:25.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.464+0000 7fcf03fff640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:25.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.464+0000 7fcf03fff640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:25.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.464+0000 7fcf03fff640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T18:16:25.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.467+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf100500c0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.467+0000 7fcf03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:25.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.467+0000 7fcf03fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:25.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.676+0000 7fcf21fe6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.676+0000 7fcf21fe6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.676+0000 7fcf21fe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:25.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.676+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf1c1dbbb0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.764+0000 7fcf03fff640 1 -- 172.21.15.87:0/4248902108 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fcf100521c0 con 0x7fcf1c0b4d60 2024-03-25T18:16:25.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.764+0000 7fcf03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:25.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.764+0000 7fcf03fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:25.765 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:25.765 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:25.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 10 librados: watch_flush enter 2024-03-25T18:16:25.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 10 librados: watch_flush exit 2024-03-25T18:16:25.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T18:16:25.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 10 client.4303.objecter successfully canceled tick 2024-03-25T18:16:25.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 >> v1:172.21.15.171:6800/3417775508 conn(0x7fcf0806e160 legacy=0x7fcf08070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 10 monclient: shutdown 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 >> v1:172.21.15.171:6789/0 conn(0x7fcf1c0b4d60 legacy=0x7fcf1c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 shutdown_connections 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 >> 172.21.15.87:0/4248902108 conn(0x7fcf1c0617a0 msgr2=0x7fcf1c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 shutdown_connections 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 -- 172.21.15.87:0/4248902108 wait complete. 2024-03-25T18:16:25.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:25.766+0000 7fcf21fe6640 1 librados: shutdown 2024-03-25T18:16:25.779 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:32.585 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:32.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.699+0000 7fe3073ef640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.699+0000 7fe3073ef640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.699+0000 7fe3073ef640 10 monclient: monmap: 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:32.699997+0000 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:32.699997+0000 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:32.700 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:32.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.699+0000 7fe3073ef640 1 Processor -- start 2024-03-25T18:16:32.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.700+0000 7fe3073ef640 1 -- start start 2024-03-25T18:16:32.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.700+0000 7fe3073ef640 10 monclient: init 2024-03-25T18:16:32.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:32.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe30006c380 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:32.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe3000b12d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:32.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe3000aedf0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:32.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000b16f0 con 0x7fe3000aedf0 2024-03-25T18:16:32.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000adfe0 con 0x7fe30006c380 2024-03-25T18:16:32.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000add00 con 0x7fe3000b12d0 2024-03-25T18:16:32.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.701+0000 7fe3073ef640 10 monclient(hunting): authenticate will time out at 7907.996118s 2024-03-25T18:16:32.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.714+0000 7fe30518b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe30006c380 0x7fe3000b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:37846/0 (socket says 172.21.15.87:37846) 2024-03-25T18:16:32.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.714+0000 7fe30518b640 1 -- 172.21.15.87:0/3245574208 learned_addr learned my addr 172.21.15.87:0/3245574208 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019026875 0 0) 0x7fe3000adfe0 con 0x7fe30006c380 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:16:32.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.723+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec003520 con 0x7fe30006c380 2024-03-25T18:16:32.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849671788 0 0) 0x7fe2ec003520 con 0x7fe30006c380 2024-03-25T18:16:32.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 >> v1:172.21.15.87:6790/0 conn(0x7fe3000b12d0 legacy=0x7fe3000ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 >> v1:172.21.15.87:6789/0 conn(0x7fe3000aedf0 legacy=0x7fe30014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000ae2c0 con 0x7fe30006c380 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe2fc001690 con 0x7fe30006c380 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe2f7fff640 10 monclient: dump: 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.731 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe3073ef640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:16:32.732 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.729+0000 7fe3073ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2fc0017f0 con 0x7fe30006c380 2024-03-25T18:16:32.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 1 -- 172.21.15.87:0/3245574208 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe2fc001b80 con 0x7fe30006c380 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe3073ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe3073ef640 4 set_mon_vals no callback set 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.734+0000 7fe2f7fff640 10 monclient: dump: 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:32.740 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 10 monclient: shutdown 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 1 -- 172.21.15.87:0/3245574208 >> v1:172.21.15.171:6789/0 conn(0x7fe30006c380 legacy=0x7fe3000b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 1 -- 172.21.15.87:0/3245574208 shutdown_connections 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 1 -- 172.21.15.87:0/3245574208 >> 172.21.15.87:0/3245574208 conn(0x7fe3000b5d10 msgr2=0x7fe3000b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 1 -- 172.21.15.87:0/3245574208 shutdown_connections 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.735+0000 7fe3073ef640 1 -- 172.21.15.87:0/3245574208 wait complete. 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.736+0000 7fe3073ef640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.736+0000 7fe3073ef640 10 monclient: monmap: 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.741 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.736+0000 7fe3073ef640 1 librados: starting msgr at 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.736+0000 7fe3073ef640 1 librados: starting objecter 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 1 Processor -- start 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 1 -- start start 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 1 librados: setting wanted keys 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 1 librados: calling monclient init 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 10 monclient: init 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.737+0000 7fe3073ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe30006c380 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe3000aedf0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe3000b12d0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:32.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30006c8e0 con 0x7fe3000aedf0 2024-03-25T18:16:32.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3001c4460 con 0x7fe3000b12d0 2024-03-25T18:16:32.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3001c4740 con 0x7fe30006c380 2024-03-25T18:16:32.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.738+0000 7fe3073ef640 10 monclient(hunting): authenticate will time out at 7908.033062s 2024-03-25T18:16:32.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.746+0000 7fe30598c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe3000b12d0 0x7fe3001c20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:37848/0 (socket says 172.21.15.87:37848) 2024-03-25T18:16:32.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.746+0000 7fe30598c640 1 -- 172.21.15.87:0/1738710041 learned_addr learned my addr 172.21.15.87:0/1738710041 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:32.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.747+0000 7fe2f5ffb640 10 client.?.objecter ms_handle_connect 0x7fe3000b12d0 2024-03-25T18:16:32.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.747+0000 7fe2f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213369459 0 0) 0x7fe3001c4460 con 0x7fe3000b12d0 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:16:32.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.751+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e4004eb0 con 0x7fe3000b12d0 2024-03-25T18:16:32.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.753+0000 7fe2f5ffb640 10 client.?.objecter ms_handle_connect 0x7fe30006c380 2024-03-25T18:16:32.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.753+0000 7fe2f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.755+0000 7fe2f5ffb640 10 client.?.objecter ms_handle_connect 0x7fe3000aedf0 2024-03-25T18:16:32.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.755+0000 7fe2f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2023982182 0 0) 0x7fe2e4004eb0 con 0x7fe3000b12d0 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3001c4460 con 0x7fe3000b12d0 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe2f8001cd0 con 0x7fe3000b12d0 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:32.759 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:32.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.758+0000 7fe2f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540306896 0 0) 0x7fe3001c4740 con 0x7fe30006c380 2024-03-25T18:16:32.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.766+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e4004eb0 con 0x7fe30006c380 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2047350897 0 0) 0x7fe3001c4460 con 0x7fe3000b12d0 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 >> v1:172.21.15.87:6790/0 conn(0x7fe30006c380 legacy=0x7fe3001dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 >> v1:172.21.15.87:6789/0 conn(0x7fe3000aedf0 legacy=0x7fe3001c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 1 monclient: found mon.b 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3001db680 con 0x7fe3000b12d0 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe2f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 10 monclient: _renew_subs 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3001c0ef0 con 0x7fe3000b12d0 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 10 monclient: _renew_subs 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3001c4d40 con 0x7fe3000b12d0 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.767+0000 7fe3073ef640 1 librados: init done 2024-03-25T18:16:32.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.768+0000 7fe3073ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.768+0000 7fe3073ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.768+0000 7fe3073ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:32.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:32.768+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3000627b0 con 0x7fe3000b12d0 2024-03-25T18:16:33.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2f8003990 con 0x7fe3000b12d0 2024-03-25T18:16:33.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:33.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe2f8003de0 con 0x7fe3000b12d0 2024-03-25T18:16:33.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f5ffb640 10 monclient: dump: 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:33.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.039+0000 7fe2f77fe640 4 set_mon_vals no callback set 2024-03-25T18:16:33.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.040+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fe2f8004000 con 0x7fe3000b12d0 2024-03-25T18:16:33.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.041+0000 7fe30518b640 10 monclient: get_auth_request con 0x7fe2e406ddd0 auth_method 0 2024-03-25T18:16:33.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.041+0000 7fe2f5ffb640 10 client.4309.objecter ms_handle_connect 0x7fe2e406ddd0 2024-03-25T18:16:33.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.047+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fe2f8054b00 con 0x7fe3000b12d0 2024-03-25T18:16:33.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.047+0000 7fe2f5ffb640 10 client.4309.objecter ms_dispatch 0x7fe300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:33.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.047+0000 7fe2f5ffb640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:33.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.047+0000 7fe2f5ffb640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:33.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.047+0000 7fe2f5ffb640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T18:16:33.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.048+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2f8050610 con 0x7fe3000b12d0 2024-03-25T18:16:33.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.048+0000 7fe2f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.048+0000 7fe2f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:33.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.270+0000 7fe3073ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.270+0000 7fe3073ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.270+0000 7fe3073ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:33.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.270+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3000718c0 con 0x7fe3000b12d0 2024-03-25T18:16:33.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.292+0000 7fe2f5ffb640 1 -- 172.21.15.87:0/1738710041 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fe2f8052540 con 0x7fe3000b12d0 2024-03-25T18:16:33.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.292+0000 7fe2f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:33.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.292+0000 7fe2f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:33.294 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:33.294 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:33.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 10 librados: watch_flush enter 2024-03-25T18:16:33.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 10 librados: watch_flush exit 2024-03-25T18:16:33.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 10 client.4309.objecter successfully canceled tick 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 >> v1:172.21.15.171:6800/3417775508 conn(0x7fe2e406ddd0 legacy=0x7fe2e4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 10 monclient: shutdown 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 >> v1:172.21.15.171:6789/0 conn(0x7fe3000b12d0 legacy=0x7fe3001c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 shutdown_connections 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 >> 172.21.15.87:0/1738710041 conn(0x7fe3000b5d10 msgr2=0x7fe3000b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.294+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 shutdown_connections 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.295+0000 7fe3073ef640 1 -- 172.21.15.87:0/1738710041 wait complete. 2024-03-25T18:16:33.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:33.295+0000 7fe3073ef640 1 librados: shutdown 2024-03-25T18:16:33.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:40.115 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.228+0000 7fa30e406640 10 monclient: get_monmap_and_config 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.228+0000 7fa30e406640 10 monclient: build_initial_monmap 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.228+0000 7fa30e406640 10 monclient: monmap: 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:40.230337+0000 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:40.230337+0000 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:40.230 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:40.231 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.229+0000 7fa30e406640 1 Processor -- start 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.229+0000 7fa30e406640 1 -- start start 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.229+0000 7fa30e406640 10 monclient: init 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-c con 0x7fa308061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-b con 0x7fa3081400e0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-a con 0x7fa3081499d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308072600 con 0x7fa3081499d0 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308140c80 con 0x7fa308061830 2024-03-25T18:16:40.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308072830 con 0x7fa3081400e0 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:40.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.230+0000 7fa30e406640 10 monclient(hunting): authenticate will time out at 7915.526239s 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.245+0000 7fa30c9a3640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa3081499d0 0x7fa30814bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57330/0 (socket says 172.21.15.87:57330) 2024-03-25T18:16:40.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.246+0000 7fa30c9a3640 1 -- 172.21.15.87:0/1627106674 learned_addr learned my addr 172.21.15.87:0/1627106674 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993343947 0 0) 0x7fa308072600 con 0x7fa3081499d0 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:16:40.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.257+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0003520 con 0x7fa3081499d0 2024-03-25T18:16:40.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382404335 0 0) 0x7fa308072830 con 0x7fa3081400e0 2024-03-25T18:16:40.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:16:40.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.270+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa308072600 con 0x7fa3081400e0 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.281+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973865399 0 0) 0x7fa2f0003520 con 0x7fa3081499d0 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 >> v1:172.21.15.87:6790/0 conn(0x7fa3081400e0 legacy=0x7fa308140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 >> v1:172.21.15.171:6789/0 conn(0x7fa308061830 legacy=0x7fa30813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 1 monclient: found mon.noname-a 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:16:40.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308072b10 con 0x7fa3081499d0 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa300001a10 con 0x7fa3081499d0 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa306ffd640 10 monclient: dump: 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.285 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.286 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa30e406640 5 monclient: authenticate success, global_id 4308 2024-03-25T18:16:40.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.282+0000 7fa30e406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa300001b70 con 0x7fa3081499d0 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:40.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 1 -- 172.21.15.87:0/1627106674 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa300001f80 con 0x7fa3081499d0 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa306ffd640 10 monclient: dump: 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:40.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 4 set_mon_vals no callback set 2024-03-25T18:16:40.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 10 monclient: shutdown 2024-03-25T18:16:40.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:40.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 1 -- 172.21.15.87:0/1627106674 >> v1:172.21.15.87:6789/0 conn(0x7fa3081499d0 legacy=0x7fa30814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 1 -- 172.21.15.87:0/1627106674 shutdown_connections 2024-03-25T18:16:40.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 1 -- 172.21.15.87:0/1627106674 >> 172.21.15.87:0/1627106674 conn(0x7fa3080b0ef0 msgr2=0x7fa3080b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 1 -- 172.21.15.87:0/1627106674 shutdown_connections 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.288+0000 7fa30e406640 1 -- 172.21.15.87:0/1627106674 wait complete. 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.289+0000 7fa30e406640 10 monclient: build_initial_monmap 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.289+0000 7fa30e406640 10 monclient: monmap: 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.289+0000 7fa30e406640 1 librados: starting msgr at 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.289+0000 7fa30e406640 1 librados: starting objecter 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 1 Processor -- start 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 1 -- start start 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 1 librados: setting wanted keys 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 1 librados: calling monclient init 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 10 monclient: init 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-b con 0x7fa308061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.290+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-c con 0x7fa3081400e0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 10 monclient(hunting): picked mon.noname-a con 0x7fa3081499d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308136bc0 con 0x7fa3081499d0 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30806f640 con 0x7fa308061830 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308136e10 con 0x7fa3081400e0 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:40.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.291+0000 7fa30e406640 10 monclient(hunting): authenticate will time out at 7915.586920s 2024-03-25T18:16:40.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.302+0000 7fa30c9a3640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa3081499d0 0x7fa3080701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57380/0 (socket says 172.21.15.87:57380) 2024-03-25T18:16:40.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.302+0000 7fa30c9a3640 1 -- 172.21.15.87:0/161027750 learned_addr learned my addr 172.21.15.87:0/161027750 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:40.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.304+0000 7fa304ff9640 10 client.?.objecter ms_handle_connect 0x7fa3081400e0 2024-03-25T18:16:40.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.304+0000 7fa304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.308+0000 7fa304ff9640 10 client.?.objecter ms_handle_connect 0x7fa3081499d0 2024-03-25T18:16:40.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.308+0000 7fa304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.308+0000 7fa304ff9640 10 client.?.objecter ms_handle_connect 0x7fa308061830 2024-03-25T18:16:40.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.308+0000 7fa304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602102142 0 0) 0x7fa30806f640 con 0x7fa308061830 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:16:40.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.321+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4004eb0 con 0x7fa308061830 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664261695 0 0) 0x7fa308136bc0 con 0x7fa3081499d0 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:16:40.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.323+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa30806f640 con 0x7fa3081499d0 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098466749 0 0) 0x7fa308136e10 con 0x7fa3081400e0 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:16:40.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.329+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa308136bc0 con 0x7fa3081400e0 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2934163995 0 0) 0x7fa2f4004eb0 con 0x7fa308061830 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa308136e10 con 0x7fa308061830 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa2f80018c0 con 0x7fa308061830 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.332 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.330+0000 7fa304ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77133989 0 0) 0x7fa30806f640 con 0x7fa3081499d0 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f4005b20 con 0x7fa3081499d0 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa300001e80 con 0x7fa3081499d0 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.334+0000 7fa304ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.336 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.337 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.337 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102698684 0 0) 0x7fa308136bc0 con 0x7fa3081400e0 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa30806f640 con 0x7fa3081400e0 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa2fc0019b0 con 0x7fa3081400e0 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.356 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2047891725 0 0) 0x7fa308136e10 con 0x7fa308061830 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 >> v1:172.21.15.87:6790/0 conn(0x7fa3081400e0 legacy=0x7fa30806fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 >> v1:172.21.15.87:6789/0 conn(0x7fa3081499d0 legacy=0x7fa3080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 monclient: found mon.b 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308134a20 con 0x7fa308061830 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:40.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 10 monclient: _renew_subs 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3081db590 con 0x7fa308061830 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 10 monclient: _renew_subs 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3081360c0 con 0x7fa308061830 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.353+0000 7fa30e406640 1 librados: init done 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.354+0000 7fa30e406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.354+0000 7fa30e406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.354+0000 7fa30e406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:40.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.354+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080627b0 con 0x7fa308061830 2024-03-25T18:16:40.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f80035f0 con 0x7fa308061830 2024-03-25T18:16:40.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:40.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa2f80018c0 con 0x7fa308061830 2024-03-25T18:16:40.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:40.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa304ff9640 10 monclient: dump: 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:40.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.646+0000 7fa3067fc640 4 set_mon_vals no callback set 2024-03-25T18:16:40.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.647+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa2f80195d0 con 0x7fa308061830 2024-03-25T18:16:40.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.648+0000 7fa3077fe640 10 monclient: get_auth_request con 0x7fa2f406e4a0 auth_method 0 2024-03-25T18:16:40.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.648+0000 7fa304ff9640 10 client.4315.objecter ms_handle_connect 0x7fa2f406e4a0 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.652+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fa2f8053290 con 0x7fa308061830 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.652+0000 7fa304ff9640 10 client.4315.objecter ms_dispatch 0x7fa308000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.652+0000 7fa304ff9640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.652+0000 7fa304ff9640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.652+0000 7fa304ff9640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T18:16:40.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.653+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2f804ebd0 con 0x7fa308061830 2024-03-25T18:16:40.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.653+0000 7fa304ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:40.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.653+0000 7fa304ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:40.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.904+0000 7fa30e406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.904+0000 7fa30e406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.904+0000 7fa30e406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:40.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.904+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3081db720 con 0x7fa308061830 2024-03-25T18:16:40.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.939+0000 7fa304ff9640 1 -- 172.21.15.87:0/161027750 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fa2f8050cd0 con 0x7fa308061830 2024-03-25T18:16:40.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.939+0000 7fa304ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:40.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.939+0000 7fa304ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:40.940 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:40.940 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.941+0000 7fa30e406640 10 librados: watch_flush enter 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.941+0000 7fa30e406640 10 librados: watch_flush exit 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 10 client.4315.objecter successfully canceled tick 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa2f406e4a0 legacy=0x7fa2f4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 10 monclient: shutdown 2024-03-25T18:16:40.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 >> v1:172.21.15.171:6789/0 conn(0x7fa308061830 legacy=0x7fa308133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 shutdown_connections 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 >> 172.21.15.87:0/161027750 conn(0x7fa3080b0ef0 msgr2=0x7fa308140570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 shutdown_connections 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 -- 172.21.15.87:0/161027750 wait complete. 2024-03-25T18:16:40.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:40.942+0000 7fa30e406640 1 librados: shutdown 2024-03-25T18:16:40.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:41.424 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:29.137 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.469 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:29.148 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.475 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:29.154 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.477 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:29.190 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.501 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:29.214 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.506 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:29.185 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.543 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:29.256 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.556 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:29.270 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:41.591 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:29.304 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:47.760 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:47.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.872+0000 7f09d8ba3640 10 monclient: get_monmap_and_config 2024-03-25T18:16:47.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.872+0000 7f09d8ba3640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.872+0000 7f09d8ba3640 10 monclient: monmap: 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:47.873617+0000 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:47.873617+0000 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:47.874 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.873+0000 7f09d8ba3640 1 Processor -- start 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.873+0000 7f09d8ba3640 1 -- start start 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.873+0000 7f09d8ba3640 10 monclient: init 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f09d40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f09d413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f09d41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4142520 con 0x7f09d41430c0 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b28a0 con 0x7f09d413f710 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b2a30 con 0x7f09d40a98f0 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.876 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.874+0000 7f09d8ba3640 10 monclient(hunting): authenticate will time out at 7923.169123s 2024-03-25T18:16:47.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.884+0000 7f09d259c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f09d40a98f0 0x7f09d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:56214/0 (socket says 172.21.15.87:56214) 2024-03-25T18:16:47.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.884+0000 7f09d259c640 1 -- 172.21.15.87:0/1842773451 learned_addr learned my addr 172.21.15.87:0/1842773451 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506570923 0 0) 0x7f09d40b28a0 con 0x7f09d413f710 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:16:47.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.890+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0003520 con 0x7f09d413f710 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025411731 0 0) 0x7f09d40b2a30 con 0x7f09d40a98f0 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.895 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:16:47.895 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.893+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d40b28a0 con 0x7f09d40a98f0 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156690059 0 0) 0x7f09d4142520 con 0x7f09d41430c0 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:16:47.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.898+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d40b2a30 con 0x7f09d41430c0 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1522940867 0 0) 0x7f09c0003520 con 0x7f09d413f710 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 >> v1:172.21.15.87:6790/0 conn(0x7f09d40a98f0 legacy=0x7f09d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 >> v1:172.21.15.87:6789/0 conn(0x7f09d41430c0 legacy=0x7f09d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40b44c0 con 0x7f09d413f710 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f09bc001690 con 0x7f09d413f710 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d8ba3640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d8ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d1d9b640 10 monclient: dump: 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:47.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.902+0000 7f09d8ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc0017f0 con 0x7f09d413f710 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 1 -- 172.21.15.87:0/1842773451 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f09bc001c00 con 0x7f09d413f710 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d1d9b640 10 monclient: dump: 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:47.908 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d8ba3640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:47.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.907+0000 7f09d8ba3640 4 set_mon_vals no callback set 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 10 monclient: shutdown 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1842773451 >> v1:172.21.15.171:6789/0 conn(0x7f09d413f710 legacy=0x7f09d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1842773451 shutdown_connections 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1842773451 >> 172.21.15.87:0/1842773451 conn(0x7f09d40b7ea0 msgr2=0x7f09d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1842773451 shutdown_connections 2024-03-25T18:16:47.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.908+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1842773451 wait complete. 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.909+0000 7f09d8ba3640 10 monclient: build_initial_monmap 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.909+0000 7f09d8ba3640 10 monclient: monmap: 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.909+0000 7f09d8ba3640 1 librados: starting msgr at 2024-03-25T18:16:47.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.909+0000 7f09d8ba3640 1 librados: starting objecter 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 Processor -- start 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 -- start start 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 librados: setting wanted keys 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 librados: calling monclient init 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient: init 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f09d40a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f09d413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f09d41430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4138250 con 0x7f09d40a98f0 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4137d50 con 0x7f09d413f710 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4137ee0 con 0x7f09d41430c0 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:47.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.910+0000 7f09d8ba3640 10 monclient(hunting): authenticate will time out at 7923.205604s 2024-03-25T18:16:47.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.915+0000 7f09c9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f09d413f710 0x7f09d41d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:54558/0 (socket says 172.21.15.87:54558) 2024-03-25T18:16:47.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.915+0000 7f09c9d9b640 1 -- 172.21.15.87:0/1281690760 learned_addr learned my addr 172.21.15.87:0/1281690760 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:47.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.917+0000 7f09cb7fe640 10 client.?.objecter ms_handle_connect 0x7f09d413f710 2024-03-25T18:16:47.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.917+0000 7f09cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52785325 0 0) 0x7f09d4137d50 con 0x7f09d413f710 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:16:47.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.922+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4004eb0 con 0x7f09d413f710 2024-03-25T18:16:47.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.925+0000 7f09cb7fe640 10 client.?.objecter ms_handle_connect 0x7f09d41430c0 2024-03-25T18:16:47.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.925+0000 7f09cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.931+0000 7f09cb7fe640 10 client.?.objecter ms_handle_connect 0x7f09d40a98f0 2024-03-25T18:16:47.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.931+0000 7f09cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:47.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442971001 0 0) 0x7f09d4137ee0 con 0x7f09d41430c0 2024-03-25T18:16:47.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:16:47.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b40051d0 con 0x7f09d41430c0 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620000250 0 0) 0x7f09d4138250 con 0x7f09d40a98f0 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.940+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d4137ee0 con 0x7f09d40a98f0 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135718759 0 0) 0x7f09b4004eb0 con 0x7f09d413f710 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d4138250 con 0x7f09d413f710 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f09bc001b80 con 0x7f09d413f710 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:47.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:47.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.941+0000 7f09cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 331233289 0 0) 0x7f09d4138250 con 0x7f09d413f710 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 >> v1:172.21.15.87:6790/0 conn(0x7f09d41430c0 legacy=0x7f09d41d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 >> v1:172.21.15.87:6789/0 conn(0x7f09d40a98f0 legacy=0x7f09d4137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 1 monclient: found mon.b 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40b7730 con 0x7f09d413f710 2024-03-25T18:16:47.950 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 10 monclient: _renew_subs 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40b7920 con 0x7f09d413f710 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 10 monclient: _renew_subs 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d41e33d0 con 0x7f09d413f710 2024-03-25T18:16:47.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.949+0000 7f09d8ba3640 1 librados: init done 2024-03-25T18:16:47.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.950+0000 7f09d8ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.950+0000 7f09d8ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:47.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.950+0000 7f09d8ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:47.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:47.950+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d40627b0 con 0x7f09d413f710 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc0011f0 con 0x7f09d413f710 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f09bc003a80 con 0x7f09d413f710 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09cb7fe640 10 monclient: dump: 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:48.218 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.216+0000 7f09d159a640 4 set_mon_vals no callback set 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.217+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f09bc003ca0 con 0x7f09d413f710 2024-03-25T18:16:48.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.218+0000 7f09d259c640 10 monclient: get_auth_request con 0x7f09b406df50 auth_method 0 2024-03-25T18:16:48.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.218+0000 7f09cb7fe640 10 client.4321.objecter ms_handle_connect 0x7f09b406df50 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f09bc054840 con 0x7f09d413f710 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 10 client.4321.objecter ms_dispatch 0x7f09d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09bc050180 con 0x7f09d413f710 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:48.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.222+0000 7f09cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:48.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.444+0000 7f09d8ba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.444+0000 7f09d8ba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.444+0000 7f09d8ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:16:48.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.444+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09d413df50 con 0x7f09d413f710 2024-03-25T18:16:48.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.465+0000 7f09cb7fe640 1 -- 172.21.15.87:0/1281690760 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f09bc052280 con 0x7f09d413f710 2024-03-25T18:16:48.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.465+0000 7f09cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:48.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.465+0000 7f09cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:48.467 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:48.467 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.468+0000 7f09d8ba3640 10 librados: watch_flush enter 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.468+0000 7f09d8ba3640 10 librados: watch_flush exit 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.468+0000 7f09d8ba3640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.468+0000 7f09d8ba3640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 >> v1:172.21.15.171:6800/3417775508 conn(0x7f09b406df50 legacy=0x7f09b4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 10 monclient: shutdown 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 >> v1:172.21.15.171:6789/0 conn(0x7f09d413f710 legacy=0x7f09d41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:48.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 shutdown_connections 2024-03-25T18:16:48.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 >> 172.21.15.87:0/1281690760 conn(0x7f09d40b7ea0 msgr2=0x7f09d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:48.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 shutdown_connections 2024-03-25T18:16:48.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 -- 172.21.15.87:0/1281690760 wait complete. 2024-03-25T18:16:48.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:48.469+0000 7f09d8ba3640 1 librados: shutdown 2024-03-25T18:16:48.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:55.289 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.402+0000 7fd38368d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.402+0000 7fd38368d640 10 monclient: build_initial_monmap 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.402+0000 7fd38368d640 10 monclient: monmap: 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.403 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:16:55.403658+0000 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:16:55.403658+0000 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.403+0000 7fd38368d640 1 Processor -- start 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.403+0000 7fd38368d640 1 -- start start 2024-03-25T18:16:55.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.403+0000 7fd38368d640 10 monclient: init 2024-03-25T18:16:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-b con 0x7fd37c0a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-a con 0x7fd37c13f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-c con 0x7fd37c1430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c142520 con 0x7fd37c13f710 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c0b28a0 con 0x7fd37c1430c0 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c0b2a30 con 0x7fd37c0a98f0 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.404+0000 7fd38368d640 10 monclient(hunting): authenticate will time out at 7930.699584s 2024-03-25T18:16:55.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.408+0000 7fd381c2a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd37c1430c0 0x7fd37c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:59222/0 (socket says 172.21.15.87:59222) 2024-03-25T18:16:55.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.408+0000 7fd381c2a640 1 -- 172.21.15.87:0/4025293210 learned_addr learned my addr 172.21.15.87:0/4025293210 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:55.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253056354 0 0) 0x7fd37c0b28a0 con 0x7fd37c1430c0 2024-03-25T18:16:55.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:16:55.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.421+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd368003520 con 0x7fd37c1430c0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728433156 0 0) 0x7fd368003520 con 0x7fd37c1430c0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 >> v1:172.21.15.87:6790/0 conn(0x7fd37c0a98f0 legacy=0x7fd37c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 >> v1:172.21.15.87:6789/0 conn(0x7fd37c13f710 legacy=0x7fd37c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c0b44c0 con 0x7fd37c1430c0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd374001a10 con 0x7fd37c1430c0 2024-03-25T18:16:55.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd38368d640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd38368d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd373fff640 10 monclient: dump: 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.428+0000 7fd38368d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd374001bb0 con 0x7fd37c1430c0 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 1 -- 172.21.15.87:0/4025293210 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd374001f80 con 0x7fd37c1430c0 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd373fff640 10 monclient: dump: 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.434 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd38368d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:55.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.433+0000 7fd38368d640 4 set_mon_vals no callback set 2024-03-25T18:16:55.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 10 monclient: shutdown 2024-03-25T18:16:55.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:55.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 1 -- 172.21.15.87:0/4025293210 >> v1:172.21.15.171:6789/0 conn(0x7fd37c1430c0 legacy=0x7fd37c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 1 -- 172.21.15.87:0/4025293210 shutdown_connections 2024-03-25T18:16:55.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 1 -- 172.21.15.87:0/4025293210 >> 172.21.15.87:0/4025293210 conn(0x7fd37c0b7ea0 msgr2=0x7fd37c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 1 -- 172.21.15.87:0/4025293210 shutdown_connections 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.434+0000 7fd38368d640 1 -- 172.21.15.87:0/4025293210 wait complete. 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 10 monclient: build_initial_monmap 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 10 monclient: monmap: 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 librados: starting msgr at 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 librados: starting objecter 2024-03-25T18:16:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 Processor -- start 2024-03-25T18:16:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 -- start start 2024-03-25T18:16:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 librados: setting wanted keys 2024-03-25T18:16:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 1 librados: calling monclient init 2024-03-25T18:16:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.435+0000 7fd38368d640 10 monclient: init 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-c con 0x7fd37c0a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-b con 0x7fd37c13f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient(hunting): picked mon.noname-a con 0x7fd37c1430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c137ab0 con 0x7fd37c1430c0 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c138bd0 con 0x7fd37c13f710 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c138e40 con 0x7fd37c0a98f0 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.436+0000 7fd38368d640 10 monclient(hunting): authenticate will time out at 7930.731573s 2024-03-25T18:16:55.447 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.446+0000 7fd381429640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd37c0a98f0 0x7fd37c137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:32832/0 (socket says 172.21.15.87:32832) 2024-03-25T18:16:55.447 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.446+0000 7fd381429640 1 -- 172.21.15.87:0/3397946823 learned_addr learned my addr 172.21.15.87:0/3397946823 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:16:55.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.449+0000 7fd371ffb640 10 client.?.objecter ms_handle_connect 0x7fd37c0a98f0 2024-03-25T18:16:55.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.449+0000 7fd371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.450+0000 7fd371ffb640 10 client.?.objecter ms_handle_connect 0x7fd37c13f710 2024-03-25T18:16:55.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.450+0000 7fd371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.455+0000 7fd371ffb640 10 client.?.objecter ms_handle_connect 0x7fd37c1430c0 2024-03-25T18:16:55.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.455+0000 7fd371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240677050 0 0) 0x7fd37c138e40 con 0x7fd37c0a98f0 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:16:55.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.461+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd360004eb0 con 0x7fd37c0a98f0 2024-03-25T18:16:55.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908392804 0 0) 0x7fd37c138bd0 con 0x7fd37c13f710 2024-03-25T18:16:55.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:16:55.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.470+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37c138e40 con 0x7fd37c13f710 2024-03-25T18:16:55.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452890587 0 0) 0x7fd37c137ab0 con 0x7fd37c1430c0 2024-03-25T18:16:55.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:55.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:55.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:55.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:16:55.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.473+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37c138bd0 con 0x7fd37c1430c0 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1096417174 0 0) 0x7fd360004eb0 con 0x7fd37c0a98f0 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd37c137ab0 con 0x7fd37c0a98f0 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd378001a30 con 0x7fd37c0a98f0 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.477 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.476+0000 7fd371ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:55.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 565106786 0 0) 0x7fd37c138e40 con 0x7fd37c13f710 2024-03-25T18:16:55.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd360004eb0 con 0x7fd37c13f710 2024-03-25T18:16:55.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd36c0018b0 con 0x7fd37c13f710 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.484+0000 7fd371ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.486 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1447478883 0 0) 0x7fd37c138bd0 con 0x7fd37c1430c0 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd37c138e40 con 0x7fd37c1430c0 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd374001d20 con 0x7fd37c1430c0 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.491 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2672257042 0 0) 0x7fd37c137ab0 con 0x7fd37c0a98f0 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 >> v1:172.21.15.171:6789/0 conn(0x7fd37c13f710 legacy=0x7fd37c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 >> v1:172.21.15.87:6789/0 conn(0x7fd37c1430c0 legacy=0x7fd37c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 monclient: found mon.c 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c139120 con 0x7fd37c0a98f0 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:55.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd38368d640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd38368d640 10 monclient: _renew_subs 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd38368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.489+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c139400 con 0x7fd37c0a98f0 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.490+0000 7fd38368d640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.490+0000 7fd38368d640 10 monclient: _renew_subs 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.490+0000 7fd38368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.490+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c0b7880 con 0x7fd37c0a98f0 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.490+0000 7fd38368d640 1 librados: init done 2024-03-25T18:16:55.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.491+0000 7fd38368d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.491+0000 7fd38368d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.491+0000 7fd38368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:55.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.491+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c137810 con 0x7fd37c0a98f0 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd378001c70 con 0x7fd37c0a98f0 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd378003a60 con 0x7fd37c0a98f0 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd371ffb640 10 monclient: dump: 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:55.796 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.795+0000 7fd3737fe640 4 set_mon_vals no callback set 2024-03-25T18:16:55.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.796+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd378003c40 con 0x7fd37c0a98f0 2024-03-25T18:16:55.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.796+0000 7fd380c28640 10 monclient: get_auth_request con 0x7fd36006e290 auth_method 0 2024-03-25T18:16:55.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.796+0000 7fd371ffb640 10 client.4328.objecter ms_handle_connect 0x7fd36006e290 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd378054890 con 0x7fd37c0a98f0 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 10 client.4328.objecter ms_dispatch 0x7fd37c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3780503a0 con 0x7fd37c0a98f0 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:55.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:55.801+0000 7fd371ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:56.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.044+0000 7fd38368d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.044+0000 7fd38368d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.044+0000 7fd38368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:16:56.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.044+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd37c0627b0 con 0x7fd37c0a98f0 2024-03-25T18:16:56.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.090+0000 7fd371ffb640 1 -- 172.21.15.87:0/3397946823 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd3780522d0 con 0x7fd37c0a98f0 2024-03-25T18:16:56.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.090+0000 7fd371ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:56.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.090+0000 7fd371ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:56.091 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:16:56.091 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.092+0000 7fd38368d640 10 librados: watch_flush enter 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.092+0000 7fd38368d640 10 librados: watch_flush exit 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.092+0000 7fd38368d640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.092+0000 7fd38368d640 10 client.4328.objecter successfully canceled tick 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd36006e290 legacy=0x7fd360070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 10 monclient: shutdown 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 >> v1:172.21.15.87:6790/0 conn(0x7fd37c0a98f0 legacy=0x7fd37c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 shutdown_connections 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 >> 172.21.15.87:0/3397946823 conn(0x7fd37c0b7ea0 msgr2=0x7fd37c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 shutdown_connections 2024-03-25T18:16:56.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 -- 172.21.15.87:0/3397946823 wait complete. 2024-03-25T18:16:56.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:16:56.093+0000 7fd38368d640 1 librados: shutdown 2024-03-25T18:16:56.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:01.505 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:49.218 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.511 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:49.225 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.540 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:02:49.254 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.579 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:49.293 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.581 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:49.260 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.586 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:49.265 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.587 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:49.300 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.613 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:02:49.291 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:01.623 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:02:49.337 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.912 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:03.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.026+0000 7f522ddf6640 10 monclient: get_monmap_and_config 2024-03-25T18:17:03.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.026+0000 7f522ddf6640 10 monclient: build_initial_monmap 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.027+0000 7f522ddf6640 10 monclient: monmap: 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:03.027840+0000 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:03.027840+0000 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:03.028 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.027+0000 7f522ddf6640 1 Processor -- start 2024-03-25T18:17:03.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.027+0000 7f522ddf6640 1 -- start start 2024-03-25T18:17:03.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.027+0000 7f522ddf6640 10 monclient: init 2024-03-25T18:17:03.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-c con 0x7f52280a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-b con 0x7f522813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-a con 0x7f52281430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5228142520 con 0x7f52281430c0 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52280b28a0 con 0x7f52280a98f0 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52280b2a30 con 0x7f522813f710 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:03.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.028+0000 7f522ddf6640 10 monclient(hunting): authenticate will time out at 7938.323558s 2024-03-25T18:17:03.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.035+0000 7f52277fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f52280a98f0 0x7f52280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:42302/0 (socket says 172.21.15.87:42302) 2024-03-25T18:17:03.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.035+0000 7f52277fe640 1 -- 172.21.15.87:0/4084954643 learned_addr learned my addr 172.21.15.87:0/4084954643 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:03.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219139821 0 0) 0x7f52280b28a0 con 0x7f52280a98f0 2024-03-25T18:17:03.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:17:03.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.049+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5210003520 con 0x7f52280a98f0 2024-03-25T18:17:03.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 992007501 0 0) 0x7f5210003520 con 0x7f52280a98f0 2024-03-25T18:17:03.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 >> v1:172.21.15.87:6790/0 conn(0x7f522813f710 legacy=0x7f52281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 >> v1:172.21.15.87:6789/0 conn(0x7f52281430c0 legacy=0x7f522814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:03.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280b44c0 con 0x7f52280a98f0 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5218001690 con 0x7f52280a98f0 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f522ddf6640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f522ddf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f52267fc640 10 monclient: dump: 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.064 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.062+0000 7f522ddf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52180017f0 con 0x7f52280a98f0 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 1 -- 172.21.15.87:0/4084954643 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5218001c00 con 0x7f52280a98f0 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f522ddf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f522ddf6640 4 set_mon_vals no callback set 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.067+0000 7f52267fc640 10 monclient: dump: 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.068 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:03.069 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 10 monclient: shutdown 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 1 -- 172.21.15.87:0/4084954643 >> v1:172.21.15.171:6789/0 conn(0x7f52280a98f0 legacy=0x7f52280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 1 -- 172.21.15.87:0/4084954643 shutdown_connections 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 1 -- 172.21.15.87:0/4084954643 >> 172.21.15.87:0/4084954643 conn(0x7f52280b7ea0 msgr2=0x7f52280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 1 -- 172.21.15.87:0/4084954643 shutdown_connections 2024-03-25T18:17:03.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 1 -- 172.21.15.87:0/4084954643 wait complete. 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 10 monclient: build_initial_monmap 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.068+0000 7f522ddf6640 10 monclient: monmap: 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 librados: starting msgr at 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 librados: starting objecter 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 Processor -- start 2024-03-25T18:17:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 -- start start 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 librados: setting wanted keys 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 1 librados: calling monclient init 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.069+0000 7f522ddf6640 10 monclient: init 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-a con 0x7f52280a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-c con 0x7f522813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient(hunting): picked mon.noname-b con 0x7f52281430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522813be70 con 0x7f52280a98f0 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5228131660 con 0x7f52281430c0 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52281317f0 con 0x7f522813f710 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:03.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.070+0000 7f522ddf6640 10 monclient(hunting): authenticate will time out at 7938.365092s 2024-03-25T18:17:03.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.076+0000 7f52277fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f52280a98f0 0x7f5228130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57748/0 (socket says 172.21.15.87:57748) 2024-03-25T18:17:03.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.076+0000 7f52277fe640 1 -- 172.21.15.87:0/2381233096 learned_addr learned my addr 172.21.15.87:0/2381233096 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:03.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.079+0000 7f520ffff640 10 client.?.objecter ms_handle_connect 0x7f52281430c0 2024-03-25T18:17:03.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.079+0000 7f520ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.079+0000 7f520ffff640 10 client.?.objecter ms_handle_connect 0x7f52280a98f0 2024-03-25T18:17:03.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.079+0000 7f520ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.082+0000 7f520ffff640 10 client.?.objecter ms_handle_connect 0x7f522813f710 2024-03-25T18:17:03.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.082+0000 7f520ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841575048 0 0) 0x7f5228131660 con 0x7f52281430c0 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:17:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.095+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5214004eb0 con 0x7f52281430c0 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785227427 0 0) 0x7f52281317f0 con 0x7f522813f710 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:17:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5228131660 con 0x7f522813f710 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890969703 0 0) 0x7f522813be70 con 0x7f52280a98f0 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:17:03.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.106+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52281317f0 con 0x7f52280a98f0 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495240242 0 0) 0x7f5214004eb0 con 0x7f52281430c0 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f522813be70 con 0x7f52281430c0 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5220001b90 con 0x7f52281430c0 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 10 monclient(hunting): dump: 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.110 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.109+0000 7f520ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675792023 0 0) 0x7f5228131660 con 0x7f522813f710 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5214004eb0 con 0x7f522813f710 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f521c0018b0 con 0x7f522813f710 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.120+0000 7f520ffff640 10 monclient(hunting): dump: 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.121 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:03.122 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3438461897 0 0) 0x7f522813be70 con 0x7f52281430c0 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 >> v1:172.21.15.87:6790/0 conn(0x7f522813f710 legacy=0x7f52281d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 >> v1:172.21.15.87:6789/0 conn(0x7f52280a98f0 legacy=0x7f5228130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 1 monclient: found mon.b 2024-03-25T18:17:03.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228131980 con 0x7f52281430c0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f520ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 10 monclient: _renew_subs 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5228135c60 con 0x7f52281430c0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 10 monclient: _renew_subs 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52280b7770 con 0x7f52281430c0 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.123+0000 7f522ddf6640 1 librados: init done 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.124+0000 7f522ddf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.124+0000 7f522ddf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.124+0000 7f522ddf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.124+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52281371d0 con 0x7f52281430c0 2024-03-25T18:17:03.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5220002080 con 0x7f52281430c0 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5220003b10 con 0x7f52281430c0 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f520ffff640 10 monclient: dump: 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:03.418 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:03.419 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:03.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.416+0000 7f5225ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:03.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.417+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f5220003d30 con 0x7f52281430c0 2024-03-25T18:17:03.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.418+0000 7f52277fe640 10 monclient: get_auth_request con 0x7f521406df60 auth_method 0 2024-03-25T18:17:03.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.418+0000 7f520ffff640 10 client.4333.objecter ms_handle_connect 0x7f521406df60 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f5220053950 con 0x7f52281430c0 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 10 client.4333.objecter ms_dispatch 0x7f5228000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f522004f290 con 0x7f52281430c0 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:03.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.422+0000 7f520ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:03.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.669+0000 7f522ddf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.669+0000 7f522ddf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.669+0000 7f522ddf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:03.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.669+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5228074cc0 con 0x7f52281430c0 2024-03-25T18:17:03.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.707+0000 7f520ffff640 1 -- 172.21.15.87:0/2381233096 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f5220051390 con 0x7f52281430c0 2024-03-25T18:17:03.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.707+0000 7f520ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:03.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.707+0000 7f520ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:03.708 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:03.708 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.709+0000 7f522ddf6640 10 librados: watch_flush enter 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.709+0000 7f522ddf6640 10 librados: watch_flush exit 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.709+0000 7f522ddf6640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 10 client.4333.objecter successfully canceled tick 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 >> v1:172.21.15.171:6800/3417775508 conn(0x7f521406df60 legacy=0x7f5214070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 10 monclient: shutdown 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:03.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 >> v1:172.21.15.171:6789/0 conn(0x7f52281430c0 legacy=0x7f52281e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:03.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 shutdown_connections 2024-03-25T18:17:03.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 >> 172.21.15.87:0/2381233096 conn(0x7f52280b7ea0 msgr2=0x7f5228074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:03.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 shutdown_connections 2024-03-25T18:17:03.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 -- 172.21.15.87:0/2381233096 wait complete. 2024-03-25T18:17:03.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:03.710+0000 7f522ddf6640 1 librados: shutdown 2024-03-25T18:17:03.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:10.529 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:10.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.639+0000 7f82d5efb640 10 monclient: get_monmap_and_config 2024-03-25T18:17:10.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.639+0000 7f82d5efb640 10 monclient: build_initial_monmap 2024-03-25T18:17:10.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.639+0000 7f82d5efb640 10 monclient: monmap: 2024-03-25T18:17:10.640 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:10.640 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:10.640524+0000 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:10.640524+0000 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:10.641 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.640+0000 7f82d5efb640 1 Processor -- start 2024-03-25T18:17:10.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.640+0000 7f82d5efb640 1 -- start start 2024-03-25T18:17:10.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.640+0000 7f82d5efb640 10 monclient: init 2024-03-25T18:17:10.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-c con 0x7f82d00a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-a con 0x7f82d013f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-b con 0x7f82d01430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0142520 con 0x7f82d013f710 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b28a0 con 0x7f82d00a98f0 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b2a30 con 0x7f82d01430c0 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:10.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.641+0000 7f82d5efb640 10 monclient(hunting): authenticate will time out at 7945.936426s 2024-03-25T18:17:10.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.645+0000 7f82cffff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f82d01430c0 0x7f82d014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:51150/0 (socket says 172.21.15.87:51150) 2024-03-25T18:17:10.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.645+0000 7f82cffff640 1 -- 172.21.15.87:0/293639404 learned_addr learned my addr 172.21.15.87:0/293639404 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:10.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894356722 0 0) 0x7f82d00b2a30 con 0x7f82d01430c0 2024-03-25T18:17:10.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:17:10.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.673+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8003520 con 0x7f82d01430c0 2024-03-25T18:17:10.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502291823 0 0) 0x7f82d0142520 con 0x7f82d013f710 2024-03-25T18:17:10.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00b2a30 con 0x7f82d013f710 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302802054 0 0) 0x7f82d00b28a0 con 0x7f82d00a98f0 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:17:10.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.675+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0142520 con 0x7f82d00a98f0 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334591834 0 0) 0x7f82b8003520 con 0x7f82d01430c0 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 >> v1:172.21.15.171:6789/0 conn(0x7f82d00a98f0 legacy=0x7f82d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 >> v1:172.21.15.87:6789/0 conn(0x7f82d013f710 legacy=0x7f82d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 monclient: found mon.noname-b 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00b44c0 con 0x7f82d01430c0 2024-03-25T18:17:10.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f82c8001a10 con 0x7f82d01430c0 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82ce7fc640 10 monclient: dump: 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:10.689 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:10.690 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82d5efb640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:17:10.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.687+0000 7f82d5efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:10.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.694+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c8001b70 con 0x7f82d01430c0 2024-03-25T18:17:10.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.694+0000 7f82ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82ce7fc640 1 -- 172.21.15.87:0/293639404 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f82c8001f80 con 0x7f82d01430c0 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82ce7fc640 10 monclient: dump: 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.696 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:10.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 4 set_mon_vals no callback set 2024-03-25T18:17:10.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 10 monclient: shutdown 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 1 -- 172.21.15.87:0/293639404 >> v1:172.21.15.87:6790/0 conn(0x7f82d01430c0 legacy=0x7f82d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 1 -- 172.21.15.87:0/293639404 shutdown_connections 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.695+0000 7f82d5efb640 1 -- 172.21.15.87:0/293639404 >> 172.21.15.87:0/293639404 conn(0x7f82d00b7ea0 msgr2=0x7f82d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.696+0000 7f82d5efb640 1 -- 172.21.15.87:0/293639404 shutdown_connections 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.696+0000 7f82d5efb640 1 -- 172.21.15.87:0/293639404 wait complete. 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.696+0000 7f82d5efb640 10 monclient: build_initial_monmap 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.696+0000 7f82d5efb640 10 monclient: monmap: 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:10.700 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 librados: starting msgr at 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 librados: starting objecter 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 Processor -- start 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 -- start start 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 librados: setting wanted keys 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 1 librados: calling monclient init 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.697+0000 7f82d5efb640 10 monclient: init 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-c con 0x7f82d00a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-b con 0x7f82d013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient(hunting): picked mon.noname-a con 0x7f82d01430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:10.701 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0137ab0 con 0x7f82d01430c0 2024-03-25T18:17:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0138bd0 con 0x7f82d013f710 2024-03-25T18:17:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0138e40 con 0x7f82d00a98f0 2024-03-25T18:17:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.698+0000 7f82d5efb640 10 monclient(hunting): authenticate will time out at 7945.993603s 2024-03-25T18:17:10.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.706+0000 7f82cffff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f82d01430c0 0x7f82d01e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57438/0 (socket says 172.21.15.87:57438) 2024-03-25T18:17:10.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.706+0000 7f82cffff640 1 -- 172.21.15.87:0/3582281105 learned_addr learned my addr 172.21.15.87:0/3582281105 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:10.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.712+0000 7f82b7fff640 10 client.?.objecter ms_handle_connect 0x7f82d01430c0 2024-03-25T18:17:10.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.712+0000 7f82b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.717+0000 7f82b7fff640 10 client.?.objecter ms_handle_connect 0x7f82d00a98f0 2024-03-25T18:17:10.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.717+0000 7f82b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.719+0000 7f82b7fff640 10 client.?.objecter ms_handle_connect 0x7f82d013f710 2024-03-25T18:17:10.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.719+0000 7f82b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106581249 0 0) 0x7f82d0137ab0 con 0x7f82d01430c0 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:17:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.728+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc004eb0 con 0x7f82d01430c0 2024-03-25T18:17:10.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342392544 0 0) 0x7f82d0138bd0 con 0x7f82d013f710 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0137ab0 con 0x7f82d013f710 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904362302 0 0) 0x7f82d0138e40 con 0x7f82d00a98f0 2024-03-25T18:17:10.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:10.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:10.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.735+0000 7f82b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:10.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.736+0000 7f82b7fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:17:10.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.736+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0138bd0 con 0x7f82d00a98f0 2024-03-25T18:17:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207390024 0 0) 0x7f82d0137ab0 con 0x7f82d013f710 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d0138e40 con 0x7f82d013f710 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f82c4001970 con 0x7f82d013f710 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:10.749 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.748+0000 7f82b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262966025 0 0) 0x7f82bc004eb0 con 0x7f82d01430c0 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d0137ab0 con 0x7f82d01430c0 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f82c8001ec0 con 0x7f82d01430c0 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.750+0000 7f82b7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:10.751 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:10.752 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:10.752 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:10.752 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:10.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1300998507 0 0) 0x7f82d0138e40 con 0x7f82d013f710 2024-03-25T18:17:10.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 >> v1:172.21.15.87:6790/0 conn(0x7f82d00a98f0 legacy=0x7f82d0137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 >> v1:172.21.15.87:6789/0 conn(0x7f82d01430c0 legacy=0x7f82d01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:10.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:10.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 1 monclient: found mon.b 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0139120 con 0x7f82d013f710 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 10 monclient: _renew_subs 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d0139400 con 0x7f82d013f710 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 10 monclient: _renew_subs 2024-03-25T18:17:10.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:10.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d00b7880 con 0x7f82d013f710 2024-03-25T18:17:10.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.762+0000 7f82d5efb640 1 librados: init done 2024-03-25T18:17:10.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.763+0000 7f82d5efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:10.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.763+0000 7f82d5efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:10.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.763+0000 7f82d5efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:10.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:10.763+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0137810 con 0x7f82d013f710 2024-03-25T18:17:11.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c4003610 con 0x7f82d013f710 2024-03-25T18:17:11.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:11.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f82c4003920 con 0x7f82d013f710 2024-03-25T18:17:11.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:11.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82b7fff640 10 monclient: dump: 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:11.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.045+0000 7f82cdffb640 4 set_mon_vals no callback set 2024-03-25T18:17:11.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.046+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f82c40198f0 con 0x7f82d013f710 2024-03-25T18:17:11.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.046+0000 7f82cf7fe640 10 monclient: get_auth_request con 0x7f82bc06df20 auth_method 0 2024-03-25T18:17:11.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.047+0000 7f82b7fff640 10 client.4339.objecter ms_handle_connect 0x7f82bc06df20 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.051+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f82c40535e0 con 0x7f82d013f710 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.051+0000 7f82b7fff640 10 client.4339.objecter ms_dispatch 0x7f82d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.051+0000 7f82b7fff640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.051+0000 7f82b7fff640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.051+0000 7f82b7fff640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.052+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82c404ef20 con 0x7f82d013f710 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.052+0000 7f82b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:11.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.052+0000 7f82b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:11.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.300+0000 7f82d5efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.300+0000 7f82d5efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.300+0000 7f82d5efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:11.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.300+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d0137c60 con 0x7f82d013f710 2024-03-25T18:17:11.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.332+0000 7f82b7fff640 1 -- 172.21.15.87:0/3582281105 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f82d0137c60 con 0x7f82d013f710 2024-03-25T18:17:11.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.332+0000 7f82b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:11.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.332+0000 7f82b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:11.333 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:11.333 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:11.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.334+0000 7f82d5efb640 10 librados: watch_flush enter 2024-03-25T18:17:11.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.334+0000 7f82d5efb640 10 librados: watch_flush exit 2024-03-25T18:17:11.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.334+0000 7f82d5efb640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T18:17:11.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.334+0000 7f82d5efb640 10 client.4339.objecter successfully canceled tick 2024-03-25T18:17:11.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.334+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 >> v1:172.21.15.171:6800/3417775508 conn(0x7f82bc06df20 legacy=0x7f82bc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 10 monclient: shutdown 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 >> v1:172.21.15.171:6789/0 conn(0x7f82d013f710 legacy=0x7f82d01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 shutdown_connections 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 >> 172.21.15.87:0/3582281105 conn(0x7f82d00b7ea0 msgr2=0x7f82d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 shutdown_connections 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 -- 172.21.15.87:0/3582281105 wait complete. 2024-03-25T18:17:11.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:11.335+0000 7f82d5efb640 1 librados: shutdown 2024-03-25T18:17:11.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:18.154 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.269+0000 7fef6b46e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.269+0000 7fef6b46e640 10 monclient: build_initial_monmap 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.269+0000 7fef6b46e640 10 monclient: monmap: 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:18.271038+0000 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:18.271038+0000 2024-03-25T18:17:18.271 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:18.272 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:18.272 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:18.272 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:18.272 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:18.272 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.270+0000 7fef6b46e640 1 Processor -- start 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.270+0000 7fef6b46e640 1 -- start start 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.270+0000 7fef6b46e640 10 monclient: init 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-c con 0x7fef640a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-b con 0x7fef6413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-a con 0x7fef641430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:18.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef64142520 con 0x7fef641430c0 2024-03-25T18:17:18.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef640b28a0 con 0x7fef640a98f0 2024-03-25T18:17:18.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef640b2a30 con 0x7fef6413f710 2024-03-25T18:17:18.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:18.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.271+0000 7fef6b46e640 10 monclient(hunting): authenticate will time out at 7953.567121s 2024-03-25T18:17:18.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.279+0000 7fef6920a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef640a98f0 0x7fef640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:49958/0 (socket says 172.21.15.87:49958) 2024-03-25T18:17:18.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.279+0000 7fef6920a640 1 -- 172.21.15.87:0/1059284655 learned_addr learned my addr 172.21.15.87:0/1059284655 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872598016 0 0) 0x7fef640b28a0 con 0x7fef640a98f0 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:17:18.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.288+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef50003520 con 0x7fef640a98f0 2024-03-25T18:17:18.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878561938 0 0) 0x7fef640b2a30 con 0x7fef6413f710 2024-03-25T18:17:18.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:17:18.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.293+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef640b28a0 con 0x7fef6413f710 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750389845 0 0) 0x7fef50003520 con 0x7fef640a98f0 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 >> v1:172.21.15.87:6790/0 conn(0x7fef6413f710 legacy=0x7fef641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 >> v1:172.21.15.87:6789/0 conn(0x7fef641430c0 legacy=0x7fef6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 monclient: found mon.noname-c 2024-03-25T18:17:18.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef640b44c0 con 0x7fef640a98f0 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fef60001690 con 0x7fef640a98f0 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef6b46e640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef6b46e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef5bfff640 10 monclient: dump: 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.300 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:18.301 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:18.301 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:18.301 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:18.301 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.297+0000 7fef6b46e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef600017f0 con 0x7fef640a98f0 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 1 -- 172.21.15.87:0/1059284655 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fef60001c00 con 0x7fef640a98f0 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef5bfff640 10 monclient: dump: 2024-03-25T18:17:18.310 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef6b46e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:18.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.308+0000 7fef6b46e640 4 set_mon_vals no callback set 2024-03-25T18:17:18.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 10 monclient: shutdown 2024-03-25T18:17:18.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:18.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 1 -- 172.21.15.87:0/1059284655 >> v1:172.21.15.171:6789/0 conn(0x7fef640a98f0 legacy=0x7fef640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 1 -- 172.21.15.87:0/1059284655 shutdown_connections 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 1 -- 172.21.15.87:0/1059284655 >> 172.21.15.87:0/1059284655 conn(0x7fef640b7ea0 msgr2=0x7fef640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 1 -- 172.21.15.87:0/1059284655 shutdown_connections 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.309+0000 7fef6b46e640 1 -- 172.21.15.87:0/1059284655 wait complete. 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 10 monclient: build_initial_monmap 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 10 monclient: monmap: 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:18.314 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 librados: starting msgr at 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 librados: starting objecter 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 Processor -- start 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 -- start start 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 librados: setting wanted keys 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 1 librados: calling monclient init 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.310+0000 7fef6b46e640 10 monclient: init 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-a con 0x7fef640a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-b con 0x7fef6413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient(hunting): picked mon.noname-c con 0x7fef641430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef641d7830 con 0x7fef640a98f0 2024-03-25T18:17:18.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef641d7500 con 0x7fef6413f710 2024-03-25T18:17:18.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef64138c70 con 0x7fef641430c0 2024-03-25T18:17:18.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:18.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.311+0000 7fef6b46e640 10 monclient(hunting): authenticate will time out at 7953.607446s 2024-03-25T18:17:18.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.318+0000 7fef68a09640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef6413f710 0x7fef64133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:49982/0 (socket says 172.21.15.87:49982) 2024-03-25T18:17:18.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.318+0000 7fef68a09640 1 -- 172.21.15.87:0/1558796265 learned_addr learned my addr 172.21.15.87:0/1558796265 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:18.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.318+0000 7fef59ffb640 10 client.?.objecter ms_handle_connect 0x7fef6413f710 2024-03-25T18:17:18.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.318+0000 7fef59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.321+0000 7fef59ffb640 10 client.?.objecter ms_handle_connect 0x7fef641430c0 2024-03-25T18:17:18.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.321+0000 7fef59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.325+0000 7fef59ffb640 10 client.?.objecter ms_handle_connect 0x7fef640a98f0 2024-03-25T18:17:18.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.325+0000 7fef59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396886415 0 0) 0x7fef641d7500 con 0x7fef6413f710 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:17:18.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.326+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef48004eb0 con 0x7fef6413f710 2024-03-25T18:17:18.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451248976 0 0) 0x7fef64138c70 con 0x7fef641430c0 2024-03-25T18:17:18.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:18.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:18.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:18.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:17:18.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.329+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef641d7500 con 0x7fef641430c0 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001392342 0 0) 0x7fef48004eb0 con 0x7fef6413f710 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef64138c70 con 0x7fef6413f710 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fef54001a10 con 0x7fef6413f710 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:18.336 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.334+0000 7fef59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 19507765 0 0) 0x7fef64138c70 con 0x7fef6413f710 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 >> v1:172.21.15.87:6790/0 conn(0x7fef641430c0 legacy=0x7fef641e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 >> v1:172.21.15.87:6789/0 conn(0x7fef640a98f0 legacy=0x7fef641d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 1 monclient: found mon.b 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64138e00 con 0x7fef6413f710 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 10 monclient: _renew_subs 2024-03-25T18:17:18.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef641346b0 con 0x7fef6413f710 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 10 monclient: _renew_subs 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.343+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef64137fc0 con 0x7fef6413f710 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.344+0000 7fef6b46e640 1 librados: init done 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.345+0000 7fef6b46e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.345+0000 7fef6b46e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.345+0000 7fef6b46e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.345+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef641d7690 con 0x7fef6413f710 2024-03-25T18:17:18.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.631+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef54001c50 con 0x7fef6413f710 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fef540038c0 con 0x7fef6413f710 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef5b7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.632+0000 7fef59ffb640 10 monclient: dump: 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:18.634 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.633+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fef54003ae0 con 0x7fef6413f710 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.633+0000 7fef6920a640 10 monclient: get_auth_request con 0x7fef4806dcc0 auth_method 0 2024-03-25T18:17:18.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.633+0000 7fef59ffb640 10 client.4345.objecter ms_handle_connect 0x7fef4806dcc0 2024-03-25T18:17:18.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.639+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fef54054710 con 0x7fef6413f710 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.639+0000 7fef59ffb640 10 client.4345.objecter ms_dispatch 0x7fef64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.639+0000 7fef59ffb640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.639+0000 7fef59ffb640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.639+0000 7fef59ffb640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.640+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef54050220 con 0x7fef6413f710 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.640+0000 7fef59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:18.641 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.640+0000 7fef59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:18.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.888+0000 7fef6b46e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:18.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.888+0000 7fef6b46e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:18.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.888+0000 7fef6b46e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:17:18.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.888+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef641d79c0 con 0x7fef6413f710 2024-03-25T18:17:18.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.930+0000 7fef59ffb640 1 -- 172.21.15.87:0/1558796265 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fef54052150 con 0x7fef6413f710 2024-03-25T18:17:18.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.930+0000 7fef59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:18.932 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.930+0000 7fef59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:18.932 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:18.932 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 10 librados: watch_flush enter 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 10 librados: watch_flush exit 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 10 client.4345.objecter successfully canceled tick 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 >> v1:172.21.15.171:6800/3417775508 conn(0x7fef4806dcc0 legacy=0x7fef48070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 10 monclient: shutdown 2024-03-25T18:17:18.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 >> v1:172.21.15.171:6789/0 conn(0x7fef6413f710 legacy=0x7fef64133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 shutdown_connections 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.932+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 >> 172.21.15.87:0/1558796265 conn(0x7fef640b7ea0 msgr2=0x7fef64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.933+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 shutdown_connections 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.933+0000 7fef6b46e640 1 -- 172.21.15.87:0/1558796265 wait complete. 2024-03-25T18:17:18.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:18.933+0000 7fef6b46e640 1 librados: shutdown 2024-03-25T18:17:18.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:21.521 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:09.235 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.569 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:09.283 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.604 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:09.318 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.637 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:09.350 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.638 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:09.316 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.642 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:09.320 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.649 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:09.362 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.682 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:09.361 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.691 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:09.404 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.752 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.865+0000 7fce91833640 10 monclient: get_monmap_and_config 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.865+0000 7fce91833640 10 monclient: build_initial_monmap 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.865+0000 7fce91833640 10 monclient: monmap: 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:25.866324+0000 2024-03-25T18:17:25.866 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:25.866324+0000 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:25.867 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 1 Processor -- start 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 1 -- start start 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient: init 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient(hunting): picked mon.noname-c con 0x7fce8c0a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient(hunting): picked mon.noname-b con 0x7fce8c13f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient(hunting): picked mon.noname-a con 0x7fce8c1430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c142520 con 0x7fce8c1430c0 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0b28a0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0b2a30 con 0x7fce8c13f710 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:25.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.866+0000 7fce91833640 10 monclient(hunting): authenticate will time out at 7961.161563s 2024-03-25T18:17:25.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.881+0000 7fce8a7fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fce8c13f710 0x7fce8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:57450/0 (socket says 172.21.15.87:57450) 2024-03-25T18:17:25.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.881+0000 7fce8a7fc640 1 -- 172.21.15.87:0/3368060207 learned_addr learned my addr 172.21.15.87:0/3368060207 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102554754 0 0) 0x7fce8c0b2a30 con 0x7fce8c13f710 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:17:25.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.887+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce74003520 con 0x7fce8c13f710 2024-03-25T18:17:25.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623185254 0 0) 0x7fce8c0b28a0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:25.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:25.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:25.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:17:25.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.892+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0b2a30 con 0x7fce8c0a98f0 2024-03-25T18:17:25.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010305091 0 0) 0x7fce8c0b2a30 con 0x7fce8c0a98f0 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 >> v1:172.21.15.87:6790/0 conn(0x7fce8c13f710 legacy=0x7fce8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 >> v1:172.21.15.87:6789/0 conn(0x7fce8c1430c0 legacy=0x7fce8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 1 monclient: found mon.noname-c 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c0b44c0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce91833640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:17:25.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.899+0000 7fce91833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.904+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fce7c001630 con 0x7fce8c0a98f0 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.904+0000 7fce89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.904+0000 7fce89ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.904+0000 7fce89ffb640 10 monclient: dump: 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:25.905 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.904+0000 7fce91833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:25.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce7c0018c0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 1 -- 172.21.15.87:0/3368060207 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fce7c001bd0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce91833640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce91833640 4 set_mon_vals no callback set 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.922+0000 7fce89ffb640 10 monclient: dump: 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:25.923 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:25.924 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:25.924 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:25.924 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 10 monclient: shutdown 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 1 -- 172.21.15.87:0/3368060207 >> v1:172.21.15.171:6789/0 conn(0x7fce8c0a98f0 legacy=0x7fce8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 1 -- 172.21.15.87:0/3368060207 shutdown_connections 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 1 -- 172.21.15.87:0/3368060207 >> 172.21.15.87:0/3368060207 conn(0x7fce8c0b7ea0 msgr2=0x7fce8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 1 -- 172.21.15.87:0/3368060207 shutdown_connections 2024-03-25T18:17:25.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.923+0000 7fce91833640 1 -- 172.21.15.87:0/3368060207 wait complete. 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 10 monclient: build_initial_monmap 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 10 monclient: monmap: 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 librados: starting msgr at 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 librados: starting objecter 2024-03-25T18:17:25.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 Processor -- start 2024-03-25T18:17:25.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 -- start start 2024-03-25T18:17:25.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 librados: setting wanted keys 2024-03-25T18:17:25.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 1 librados: calling monclient init 2024-03-25T18:17:25.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.924+0000 7fce91833640 10 monclient: init 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient(hunting): picked mon.noname-b con 0x7fce8c0a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient(hunting): picked mon.noname-a con 0x7fce8c13f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient(hunting): picked mon.noname-c con 0x7fce8c1430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c1dac80 con 0x7fce8c13f710 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0abe70 con 0x7fce8c0a98f0 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c1da880 con 0x7fce8c1430c0 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:25.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.925+0000 7fce91833640 10 monclient(hunting): authenticate will time out at 7961.220790s 2024-03-25T18:17:25.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.929+0000 7fce8affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fce8c0a98f0 0x7fce8c1db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:54534/0 (socket says 172.21.15.87:54534) 2024-03-25T18:17:25.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.929+0000 7fce8affd640 1 -- 172.21.15.87:0/1702568322 learned_addr learned my addr 172.21.15.87:0/1702568322 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:25.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.930+0000 7fce73fff640 10 client.?.objecter ms_handle_connect 0x7fce8c0a98f0 2024-03-25T18:17:25.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.930+0000 7fce73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 client.?.objecter ms_handle_connect 0x7fce8c13f710 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472393647 0 0) 0x7fce8c0abe70 con 0x7fce8c0a98f0 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:17:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.940+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce78004eb0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 client.?.objecter ms_handle_connect 0x7fce8c1430c0 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893980889 0 0) 0x7fce8c1dac80 con 0x7fce8c13f710 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:25.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:17:25.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.951+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0abe70 con 0x7fce8c13f710 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923109073 0 0) 0x7fce8c1da880 con 0x7fce8c1430c0 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:17:25.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.964+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c1dac80 con 0x7fce8c1430c0 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3678025146 0 0) 0x7fce8c0abe70 con 0x7fce8c13f710 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8c1da880 con 0x7fce8c13f710 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fce80001950 con 0x7fce8c13f710 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 10 monclient(hunting): dump: 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:25.969 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:25.970 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.968+0000 7fce73fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327381720 0 0) 0x7fce78004eb0 con 0x7fce8c0a98f0 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8c0abe70 con 0x7fce8c0a98f0 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fce7c001730 con 0x7fce8c0a98f0 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.970+0000 7fce73fff640 10 monclient(hunting): dump: 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:25.971 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:25.972 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:25.972 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:25.972 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:25.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208623076 0 0) 0x7fce8c1da880 con 0x7fce8c13f710 2024-03-25T18:17:25.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 >> v1:172.21.15.87:6790/0 conn(0x7fce8c1430c0 legacy=0x7fce8c1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:25.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 >> v1:172.21.15.171:6789/0 conn(0x7fce8c0a98f0 legacy=0x7fce8c1db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:25.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:25.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 1 monclient: found mon.a 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c137b80 con 0x7fce8c13f710 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 5 monclient: authenticate success, global_id 4344 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 10 monclient: _renew_subs 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c138b00 con 0x7fce8c13f710 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 10 monclient: _renew_subs 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c1e34b0 con 0x7fce8c13f710 2024-03-25T18:17:25.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.980+0000 7fce91833640 1 librados: init done 2024-03-25T18:17:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.981+0000 7fce91833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.982+0000 7fce91833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.982+0000 7fce91833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:25.982+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c0627b0 con 0x7fce8c13f710 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce80001d60 con 0x7fce8c13f710 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fce80004980 con 0x7fce8c13f710 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce73fff640 10 monclient: dump: 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:26.288 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.287+0000 7fce897fa640 4 set_mon_vals no callback set 2024-03-25T18:17:26.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.288+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fce80004ba0 con 0x7fce8c13f710 2024-03-25T18:17:26.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.289+0000 7fce8affd640 10 monclient: get_auth_request con 0x7fce7806df60 auth_method 0 2024-03-25T18:17:26.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.289+0000 7fce73fff640 10 client.4344.objecter ms_handle_connect 0x7fce7806df60 2024-03-25T18:17:26.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.299+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fce80054790 con 0x7fce8c13f710 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.299+0000 7fce73fff640 10 client.4344.objecter ms_dispatch 0x7fce8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.299+0000 7fce73fff640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.299+0000 7fce73fff640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.300+0000 7fce73fff640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.300+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce800502a0 con 0x7fce8c13f710 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.300+0000 7fce73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:26.301 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.300+0000 7fce73fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:26.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.510+0000 7fce91833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.510+0000 7fce91833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.510+0000 7fce91833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:26.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.510+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce8c074cc0 con 0x7fce8c13f710 2024-03-25T18:17:26.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.537+0000 7fce73fff640 1 -- 172.21.15.87:0/1702568322 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fce800521d0 con 0x7fce8c13f710 2024-03-25T18:17:26.538 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.537+0000 7fce73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:26.538 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.537+0000 7fce73fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:26.538 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:26.538 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:26.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 10 librados: watch_flush enter 2024-03-25T18:17:26.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 10 librados: watch_flush exit 2024-03-25T18:17:26.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 10 client.4344.objecter successfully canceled tick 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 >> v1:172.21.15.171:6800/3417775508 conn(0x7fce7806df60 legacy=0x7fce78070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 10 monclient: shutdown 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 >> v1:172.21.15.87:6789/0 conn(0x7fce8c13f710 legacy=0x7fce8c1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 shutdown_connections 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 >> 172.21.15.87:0/1702568322 conn(0x7fce8c0b7ea0 msgr2=0x7fce8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 shutdown_connections 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 -- 172.21.15.87:0/1702568322 wait complete. 2024-03-25T18:17:26.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:26.539+0000 7fce91833640 1 librados: shutdown 2024-03-25T18:17:26.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:33.357 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:33.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:33.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 10 monclient: build_initial_monmap 2024-03-25T18:17:33.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 10 monclient: monmap: 2024-03-25T18:17:33.523 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:33.470979+0000 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:33.470979+0000 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:33.524 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 1 Processor -- start 2024-03-25T18:17:33.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 1 -- start start 2024-03-25T18:17:33.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.470+0000 7f8e51f0e640 10 monclient: init 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c0b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4c0b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0af940 con 0x7f8e4c14be90 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0b0a70 con 0x7f8e4c0b1730 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0b0d50 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.471+0000 7f8e51f0e640 10 monclient(hunting): authenticate will time out at 7968.766687s 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.481+0000 7f8e4b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f8e4c0b4d60 0x7f8e4c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:51362/0 (socket says 172.21.15.87:51362) 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.482+0000 7f8e4b7fe640 1 -- 172.21.15.87:0/3561531192 learned_addr learned my addr 172.21.15.87:0/3561531192 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.504+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504153978 0 0) 0x7f8e4c0b0d50 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.505+0000 7f8e4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.505+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.505+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.505+0000 7f8e4a7fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.505+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e34003520 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199306117 0 0) 0x7f8e4c0b0a70 con 0x7f8e4c0b1730 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.510+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c0b0d50 con 0x7f8e4c0b1730 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670760583 0 0) 0x7f8e4c0af940 con 0x7f8e4c14be90 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.513+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c0b0a70 con 0x7f8e4c14be90 2024-03-25T18:17:33.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738599875 0 0) 0x7f8e34003520 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 >> v1:172.21.15.171:6789/0 conn(0x7f8e4c0b1730 legacy=0x7f8e4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 >> v1:172.21.15.87:6789/0 conn(0x7f8e4c14be90 legacy=0x7f8e4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0b1030 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e3c001690 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e4a7fc640 10 monclient: dump: 2024-03-25T18:17:33.529 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e51f0e640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:17:33.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.519+0000 7f8e51f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:33.539 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e3c0017f0 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 1 -- 172.21.15.87:0/3561531192 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e3c001c00 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e4a7fc640 10 monclient: dump: 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.540 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.541 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.541 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e51f0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:33.541 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.539+0000 7f8e51f0e640 4 set_mon_vals no callback set 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 10 monclient: shutdown 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 1 -- 172.21.15.87:0/3561531192 >> v1:172.21.15.87:6790/0 conn(0x7f8e4c0b4d60 legacy=0x7f8e4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 1 -- 172.21.15.87:0/3561531192 shutdown_connections 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 1 -- 172.21.15.87:0/3561531192 >> 172.21.15.87:0/3561531192 conn(0x7f8e4c0617a0 msgr2=0x7f8e4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 1 -- 172.21.15.87:0/3561531192 shutdown_connections 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 1 -- 172.21.15.87:0/3561531192 wait complete. 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 10 monclient: build_initial_monmap 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.540+0000 7f8e51f0e640 10 monclient: monmap: 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:33.543 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 librados: starting msgr at 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 librados: starting objecter 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 Processor -- start 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 -- start start 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 librados: setting wanted keys 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 1 librados: calling monclient init 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.541+0000 7f8e51f0e640 10 monclient: init 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:33.544 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4c14be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c13ab20 con 0x7f8e4c0b1730 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c1d7b90 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c143030 con 0x7f8e4c14be90 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:33.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.542+0000 7f8e51f0e640 10 monclient(hunting): authenticate will time out at 7968.837280s 2024-03-25T18:17:33.551 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.550+0000 7f8e4bfff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f8e4c14be90 0x7f8e4c13b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:51416/0 (socket says 172.21.15.87:51416) 2024-03-25T18:17:33.551 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.550+0000 7f8e4bfff640 1 -- 172.21.15.87:0/4128381530 learned_addr learned my addr 172.21.15.87:0/4128381530 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:33.551 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.551+0000 7f8e33fff640 10 client.?.objecter ms_handle_connect 0x7f8e4c14be90 2024-03-25T18:17:33.551 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.551+0000 7f8e33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.557+0000 7f8e33fff640 10 client.?.objecter ms_handle_connect 0x7f8e4c0b4d60 2024-03-25T18:17:33.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.557+0000 7f8e33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.560+0000 7f8e33fff640 10 client.?.objecter ms_handle_connect 0x7f8e4c0b1730 2024-03-25T18:17:33.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.560+0000 7f8e33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:33.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775634225 0 0) 0x7f8e4c1d7b90 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.564 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.564 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.564 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:17:33.564 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.563+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38004eb0 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.565 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367184949 0 0) 0x7f8e4c143030 con 0x7f8e4c14be90 2024-03-25T18:17:33.565 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.565 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.565 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.566 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:17:33.566 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.564+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c1d7b90 con 0x7f8e4c14be90 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432839147 0 0) 0x7f8e4c13ab20 con 0x7f8e4c0b1730 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:17:33.576 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.575+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c143030 con 0x7f8e4c0b1730 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771748523 0 0) 0x7f8e4c1d7b90 con 0x7f8e4c14be90 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4c13ab20 con 0x7f8e4c14be90 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e44001c50 con 0x7f8e4c14be90 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 10 monclient(hunting): dump: 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.586 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.585+0000 7f8e33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482134682 0 0) 0x7f8e4c143030 con 0x7f8e4c0b1730 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4c1d7b90 con 0x7f8e4c0b1730 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e3c001b00 con 0x7f8e4c0b1730 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:33.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.587+0000 7f8e33fff640 10 monclient(hunting): dump: 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.589 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541328506 0 0) 0x7f8e38004eb0 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4c143030 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e40001970 con 0x7f8e4c0b4d60 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.590+0000 7f8e33fff640 10 monclient(hunting): dump: 2024-03-25T18:17:33.591 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.592 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1242944316 0 0) 0x7f8e4c1d7b90 con 0x7f8e4c0b1730 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 >> v1:172.21.15.87:6790/0 conn(0x7f8e4c14be90 legacy=0x7f8e4c13b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 >> v1:172.21.15.171:6789/0 conn(0x7f8e4c0b4d60 legacy=0x7f8e4c1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 1 monclient: found mon.a 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c1d87c0 con 0x7f8e4c0b1730 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 10 monclient: _renew_subs 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:33.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c1432e0 con 0x7f8e4c0b1730 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 10 monclient: _renew_subs 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c13b190 con 0x7f8e4c0b1730 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.597+0000 7f8e51f0e640 1 librados: init done 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.598+0000 7f8e51f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.598+0000 7f8e51f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.598+0000 7f8e51f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:33.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.598+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c1401f0 con 0x7f8e4c0b1730 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e3c001c60 con 0x7f8e4c0b1730 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e3c004c30 con 0x7f8e4c0b1730 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 10 monclient: dump: 2024-03-25T18:17:33.902 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e49ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:33.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.901+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f8e3c01ac00 con 0x7f8e4c0b1730 2024-03-25T18:17:33.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.902+0000 7f8e4affd640 10 monclient: get_auth_request con 0x7f8e3806e100 auth_method 0 2024-03-25T18:17:33.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.902+0000 7f8e33fff640 10 client.4350.objecter ms_handle_connect 0x7f8e3806e100 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f8e3c054830 con 0x7f8e4c0b1730 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 10 client.4350.objecter ms_dispatch 0x7f8e4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e3c050340 con 0x7f8e4c0b1730 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:33.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:33.907+0000 7f8e33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:34.135 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.135+0000 7f8e51f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.135+0000 7f8e51f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.135+0000 7f8e51f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:34.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.135+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e4c1d7d20 con 0x7f8e4c0b1730 2024-03-25T18:17:34.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.156+0000 7f8e33fff640 1 -- 172.21.15.87:0/4128381530 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f8e3c052270 con 0x7f8e4c0b1730 2024-03-25T18:17:34.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.156+0000 7f8e33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:34.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.156+0000 7f8e33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:34.157 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:34.157 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 10 librados: watch_flush enter 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 10 librados: watch_flush exit 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 >> v1:172.21.15.171:6800/3417775508 conn(0x7f8e3806e100 legacy=0x7f8e380705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 10 monclient: shutdown 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.158+0000 7f8e51f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 >> v1:172.21.15.87:6789/0 conn(0x7f8e4c0b1730 legacy=0x7f8e4c139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 shutdown_connections 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 >> 172.21.15.87:0/4128381530 conn(0x7f8e4c0617a0 msgr2=0x7f8e4c145330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:34.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 shutdown_connections 2024-03-25T18:17:34.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 -- 172.21.15.87:0/4128381530 wait complete. 2024-03-25T18:17:34.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:34.159+0000 7f8e51f0e640 1 librados: shutdown 2024-03-25T18:17:34.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:40.977 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:41.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.090+0000 7ff57b2ae640 10 monclient: get_monmap_and_config 2024-03-25T18:17:41.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.090+0000 7ff57b2ae640 10 monclient: build_initial_monmap 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.090+0000 7ff57b2ae640 10 monclient: monmap: 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:41.091768+0000 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:41.091768+0000 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:41.092 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.091+0000 7ff57b2ae640 1 Processor -- start 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.091+0000 7ff57b2ae640 1 -- start start 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.091+0000 7ff57b2ae640 10 monclient: init 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:41.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-b con 0x7ff5740a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-c con 0x7ff57413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-a con 0x7ff5741430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574142520 con 0x7ff5741430c0 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740b28a0 con 0x7ff57413f710 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740b2a30 con 0x7ff5740a98f0 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:41.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.092+0000 7ff57b2ae640 10 monclient(hunting): authenticate will time out at 7976.387591s 2024-03-25T18:17:41.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.103+0000 7ff57904a640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7ff5740a98f0 0x7ff5740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:33712/0 (socket says 172.21.15.87:33712) 2024-03-25T18:17:41.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.103+0000 7ff57904a640 1 -- 172.21.15.87:0/2926544240 learned_addr learned my addr 172.21.15.87:0/2926544240 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125008734 0 0) 0x7ff5740b2a30 con 0x7ff5740a98f0 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:17:41.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.112+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff560003520 con 0x7ff5740a98f0 2024-03-25T18:17:41.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818039908 0 0) 0x7ff560003520 con 0x7ff5740a98f0 2024-03-25T18:17:41.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 >> v1:172.21.15.171:6789/0 conn(0x7ff57413f710 legacy=0x7ff5741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 >> v1:172.21.15.87:6789/0 conn(0x7ff5741430c0 legacy=0x7ff57414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:41.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 monclient: found mon.noname-b 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740b44c0 con 0x7ff5740a98f0 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff570001690 con 0x7ff5740a98f0 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff56bfff640 10 monclient: dump: 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.121 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff57b2ae640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:17:41.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.119+0000 7ff57b2ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:41.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5700017f0 con 0x7ff5740a98f0 2024-03-25T18:17:41.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:41.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 1 -- 172.21.15.87:0/2926544240 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff570001c40 con 0x7ff5740a98f0 2024-03-25T18:17:41.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff57b2ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:41.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff57b2ae640 4 set_mon_vals no callback set 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.139+0000 7ff56bfff640 10 monclient: dump: 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:41.141 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.140+0000 7ff57b2ae640 10 monclient: shutdown 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.140+0000 7ff57b2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.140+0000 7ff57b2ae640 1 -- 172.21.15.87:0/2926544240 >> v1:172.21.15.87:6790/0 conn(0x7ff5740a98f0 legacy=0x7ff5740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.140+0000 7ff57b2ae640 1 -- 172.21.15.87:0/2926544240 shutdown_connections 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.140+0000 7ff57b2ae640 1 -- 172.21.15.87:0/2926544240 >> 172.21.15.87:0/2926544240 conn(0x7ff5740b7ea0 msgr2=0x7ff5740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 1 -- 172.21.15.87:0/2926544240 shutdown_connections 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 1 -- 172.21.15.87:0/2926544240 wait complete. 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 10 monclient: build_initial_monmap 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 10 monclient: monmap: 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:41.143 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 1 librados: starting msgr at 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.141+0000 7ff57b2ae640 1 librados: starting objecter 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 Processor -- start 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 -- start start 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 librados: setting wanted keys 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 librados: calling monclient init 2024-03-25T18:17:41.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient: init 2024-03-25T18:17:41.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:41.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:41.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-a con 0x7ff5740a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:41.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-c con 0x7ff57413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:41.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient(hunting): picked mon.noname-b con 0x7ff5741430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:41.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574139a40 con 0x7ff5740a98f0 2024-03-25T18:17:41.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574133f20 con 0x7ff5741430c0 2024-03-25T18:17:41.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5740ab950 con 0x7ff57413f710 2024-03-25T18:17:41.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:41.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.142+0000 7ff57b2ae640 10 monclient(hunting): authenticate will time out at 7976.437802s 2024-03-25T18:17:41.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.153+0000 7ff57984b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff5741430c0 0x7ff574133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52630/0 (socket says 172.21.15.87:52630) 2024-03-25T18:17:41.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.153+0000 7ff57984b640 1 -- 172.21.15.87:0/153674162 learned_addr learned my addr 172.21.15.87:0/153674162 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:41.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.154+0000 7ff569ffb640 10 client.?.objecter ms_handle_connect 0x7ff5741430c0 2024-03-25T18:17:41.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.154+0000 7ff569ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.159+0000 7ff569ffb640 10 client.?.objecter ms_handle_connect 0x7ff5740a98f0 2024-03-25T18:17:41.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.159+0000 7ff569ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.163+0000 7ff569ffb640 10 client.?.objecter ms_handle_connect 0x7ff57413f710 2024-03-25T18:17:41.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.163+0000 7ff569ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012605075 0 0) 0x7ff574133f20 con 0x7ff5741430c0 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:17:41.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.168+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff558004eb0 con 0x7ff5741430c0 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461997922 0 0) 0x7ff574139a40 con 0x7ff5740a98f0 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:17:41.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.170+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574133f20 con 0x7ff5740a98f0 2024-03-25T18:17:41.185 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621539911 0 0) 0x7ff5740ab950 con 0x7ff57413f710 2024-03-25T18:17:41.185 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574139a40 con 0x7ff57413f710 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2558335023 0 0) 0x7ff558004eb0 con 0x7ff5741430c0 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5740ab950 con 0x7ff5741430c0 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff56c001c90 con 0x7ff5741430c0 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.186 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.184+0000 7ff569ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:41.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799769848 0 0) 0x7ff574133f20 con 0x7ff5740a98f0 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff558004eb0 con 0x7ff5740a98f0 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff570001b00 con 0x7ff5740a98f0 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.186+0000 7ff569ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:41.188 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3224399062 0 0) 0x7ff558004eb0 con 0x7ff5740a98f0 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 >> v1:172.21.15.87:6790/0 conn(0x7ff57413f710 legacy=0x7ff574133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 >> v1:172.21.15.171:6789/0 conn(0x7ff5741430c0 legacy=0x7ff574133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 1 monclient: found mon.a 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740abc30 con 0x7ff5740a98f0 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff57b2ae640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:17:41.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff57b2ae640 10 monclient: _renew_subs 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff57b2ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.200+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5740abf10 con 0x7ff5740a98f0 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.201+0000 7ff57b2ae640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.201+0000 7ff57b2ae640 10 monclient: _renew_subs 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.201+0000 7ff57b2ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.201+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff574139370 con 0x7ff5740a98f0 2024-03-25T18:17:41.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.201+0000 7ff57b2ae640 1 librados: init done 2024-03-25T18:17:41.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.202+0000 7ff57b2ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.202+0000 7ff57b2ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.202+0000 7ff57b2ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:41.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.202+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5740627b0 con 0x7ff5740a98f0 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff570001010 con 0x7ff5740a98f0 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7ff570004bf0 con 0x7ff5740a98f0 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff569ffb640 10 monclient: dump: 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:41.502 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.501+0000 7ff56b7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.502+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7ff57001ac40 con 0x7ff5740a98f0 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.502+0000 7ff578849640 10 monclient: get_auth_request con 0x7ff55806df60 auth_method 0 2024-03-25T18:17:41.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.503+0000 7ff569ffb640 10 client.4356.objecter ms_handle_connect 0x7ff55806df60 2024-03-25T18:17:41.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7ff570054910 con 0x7ff5740a98f0 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 10 client.4356.objecter ms_dispatch 0x7ff574000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff570050250 con 0x7ff5740a98f0 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:41.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.508+0000 7ff569ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:41.704 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:29.417 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.714 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:29.428 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.745+0000 7ff57b2ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:41.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.745+0000 7ff57b2ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:41.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.745+0000 7ff57b2ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:41.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.745+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5740b7180 con 0x7ff5740a98f0 2024-03-25T18:17:41.769 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:29.481 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.772 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:29.485 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.772 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:29.451 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.772+0000 7ff569ffb640 1 -- 172.21.15.87:0/153674162 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7ff570052350 con 0x7ff5740a98f0 2024-03-25T18:17:41.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.772+0000 7ff569ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:41.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.772+0000 7ff569ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:41.773 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:41.773 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:41.774 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:29.487 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.774+0000 7ff57b2ae640 10 librados: watch_flush enter 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.774+0000 7ff57b2ae640 10 librados: watch_flush exit 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.774+0000 7ff57b2ae640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.774+0000 7ff57b2ae640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 >> v1:172.21.15.171:6800/3417775508 conn(0x7ff55806df60 legacy=0x7ff558070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 10 monclient: shutdown 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 >> v1:172.21.15.87:6789/0 conn(0x7ff5740a98f0 legacy=0x7ff574138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:41.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 shutdown_connections 2024-03-25T18:17:41.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 >> 172.21.15.87:0/153674162 conn(0x7ff5740b7ea0 msgr2=0x7ff574074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:41.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 shutdown_connections 2024-03-25T18:17:41.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 -- 172.21.15.87:0/153674162 wait complete. 2024-03-25T18:17:41.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:41.775+0000 7ff57b2ae640 1 librados: shutdown 2024-03-25T18:17:41.779 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:29.458 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:41.798 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:29.477 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:41.801 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:29.514 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.594 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.709+0000 7f5ed950e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.709+0000 7f5ed950e640 10 monclient: build_initial_monmap 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.709+0000 7f5ed950e640 10 monclient: monmap: 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:48.710289+0000 2024-03-25T18:17:48.710 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:48.710289+0000 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:48.711 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.710+0000 7f5ed950e640 1 Processor -- start 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.710+0000 7f5ed950e640 1 -- start start 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.710+0000 7f5ed950e640 10 monclient: init 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.710+0000 7f5ed950e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4142520 con 0x7f5ed41430c0 2024-03-25T18:17:48.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40b28a0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40b2a30 con 0x7f5ed413f710 2024-03-25T18:17:48.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:48.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.711+0000 7f5ed950e640 10 monclient(hunting): authenticate will time out at 7984.005929s 2024-03-25T18:17:48.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.718+0000 7f5ed2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5ed40a98f0 0x7f5ed40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:50110/0 (socket says 172.21.15.87:50110) 2024-03-25T18:17:48.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.718+0000 7f5ed2ffd640 1 -- 172.21.15.87:0/1964434096 learned_addr learned my addr 172.21.15.87:0/1964434096 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599347933 0 0) 0x7f5ed40b28a0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:17:48.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.726+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec0003520 con 0x7f5ed40a98f0 2024-03-25T18:17:48.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893072489 0 0) 0x7f5ed4142520 con 0x7f5ed41430c0 2024-03-25T18:17:48.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:48.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:48.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:48.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:17:48.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.729+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec0003840 con 0x7f5ed41430c0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.733+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395136334 0 0) 0x7f5ec0003520 con 0x7f5ed40a98f0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 >> v1:172.21.15.87:6790/0 conn(0x7f5ed413f710 legacy=0x7f5ed41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 >> v1:172.21.15.87:6789/0 conn(0x7f5ed41430c0 legacy=0x7f5ed414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40b44c0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:48.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ec4001670 con 0x7f5ed40a98f0 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed27fc640 10 monclient: dump: 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed950e640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:17:48.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.734+0000 7f5ed950e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:48.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec40017d0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:48.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 1 -- 172.21.15.87:0/1964434096 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ec4001be0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed27fc640 10 monclient: dump: 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed950e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:48.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.747+0000 7f5ed950e640 4 set_mon_vals no callback set 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 10 monclient: shutdown 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 1 -- 172.21.15.87:0/1964434096 >> v1:172.21.15.171:6789/0 conn(0x7f5ed40a98f0 legacy=0x7f5ed40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 1 -- 172.21.15.87:0/1964434096 shutdown_connections 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 1 -- 172.21.15.87:0/1964434096 >> 172.21.15.87:0/1964434096 conn(0x7f5ed40b7ea0 msgr2=0x7f5ed40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 1 -- 172.21.15.87:0/1964434096 shutdown_connections 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.748+0000 7f5ed950e640 1 -- 172.21.15.87:0/1964434096 wait complete. 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.749+0000 7f5ed950e640 10 monclient: build_initial_monmap 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.749+0000 7f5ed950e640 10 monclient: monmap: 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:48.751 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.749+0000 7f5ed950e640 1 librados: starting msgr at 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 1 librados: starting objecter 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 1 Processor -- start 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 1 -- start start 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 1 librados: setting wanted keys 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 1 librados: calling monclient init 2024-03-25T18:17:48.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.750+0000 7f5ed950e640 10 monclient: init 2024-03-25T18:17:48.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:48.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4139390 con 0x7f5ed41430c0 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4138060 con 0x7f5ed413f710 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4138580 con 0x7f5ed40a98f0 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:48.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.751+0000 7f5ed950e640 10 monclient(hunting): authenticate will time out at 7984.046122s 2024-03-25T18:17:48.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.760+0000 7f5ed37fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f5ed41430c0 0x7f5ed41e2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54674/0 (socket says 172.21.15.87:54674) 2024-03-25T18:17:48.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.760+0000 7f5ed37fe640 1 -- 172.21.15.87:0/1561929905 learned_addr learned my addr 172.21.15.87:0/1561929905 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:48.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.764+0000 7f5ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ed41430c0 2024-03-25T18:17:48.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.764+0000 7f5ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:48.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.764+0000 7f5ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ed40a98f0 2024-03-25T18:17:48.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.764+0000 7f5ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:48.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.769+0000 7f5ecb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ed413f710 2024-03-25T18:17:48.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.769+0000 7f5ecb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667540420 0 0) 0x7f5ed4138580 con 0x7f5ed40a98f0 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:17:48.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.774+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418280408 0 0) 0x7f5ed4139390 con 0x7f5ed41430c0 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:17:48.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.782+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4138580 con 0x7f5ed41430c0 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854547740 0 0) 0x7f5ed4138060 con 0x7f5ed413f710 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:17:48.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.783+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4139390 con 0x7f5ed413f710 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3552504490 0 0) 0x7f5eb4004eb0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed4138060 con 0x7f5ed40a98f0 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ec4001ae0 con 0x7f5ed40a98f0 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:48.791 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.790+0000 7f5ecb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546617777 0 0) 0x7f5ed4139390 con 0x7f5ed413f710 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb4004eb0 con 0x7f5ed413f710 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ebc001830 con 0x7f5ed413f710 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.791+0000 7f5ecb7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.793 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:48.794 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:48.794 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:48.794 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3737357956 0 0) 0x7f5ed4138580 con 0x7f5ed41430c0 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed4139390 con 0x7f5ed41430c0 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ecc001c10 con 0x7f5ed41430c0 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.794+0000 7f5ecb7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:48.795 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:48.796 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2309155405 0 0) 0x7f5ed4139390 con 0x7f5ed41430c0 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 >> v1:172.21.15.87:6790/0 conn(0x7f5ed40a98f0 legacy=0x7f5ed41dabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 >> v1:172.21.15.171:6789/0 conn(0x7f5ed413f710 legacy=0x7f5ed4132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 1 monclient: found mon.a 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4137af0 con 0x7f5ed41430c0 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:48.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 10 monclient: _renew_subs 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed4137d40 con 0x7f5ed41430c0 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 10 monclient: _renew_subs 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed41372a0 con 0x7f5ed41430c0 2024-03-25T18:17:48.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.805+0000 7f5ed950e640 1 librados: init done 2024-03-25T18:17:48.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.806+0000 7f5ed950e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:48.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.806+0000 7f5ed950e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:48.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.806+0000 7f5ed950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:48.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:48.806+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed41381f0 con 0x7f5ed41430c0 2024-03-25T18:17:49.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ecc0020e0 con 0x7f5ed41430c0 2024-03-25T18:17:49.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:49.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5ecc003b10 con 0x7f5ed41430c0 2024-03-25T18:17:49.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:49.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ecb7fe640 10 monclient: dump: 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:49.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.121+0000 7f5ed1ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:49.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.122+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f5ecc003d30 con 0x7f5ed41430c0 2024-03-25T18:17:49.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.122+0000 7f5ed2ffd640 10 monclient: get_auth_request con 0x7f5eb406e110 auth_method 0 2024-03-25T18:17:49.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.123+0000 7f5ecb7fe640 10 client.4362.objecter ms_handle_connect 0x7f5eb406e110 2024-03-25T18:17:49.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f5ecc054a20 con 0x7f5ed41430c0 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 10 client.4362.objecter ms_dispatch 0x7f5ed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.127+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ecc050360 con 0x7f5ed41430c0 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.128+0000 7f5ecb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:49.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.128+0000 7f5ecb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:49.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.360+0000 7f5ed950e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.360+0000 7f5ed950e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.360+0000 7f5ed950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:49.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.360+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed40627b0 con 0x7f5ed41430c0 2024-03-25T18:17:49.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.415+0000 7f5ecb7fe640 1 -- 172.21.15.87:0/1561929905 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f5ecc052460 con 0x7f5ed41430c0 2024-03-25T18:17:49.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.415+0000 7f5ecb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:49.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.415+0000 7f5ecb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:49.416 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:49.416 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 10 librados: watch_flush enter 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 10 librados: watch_flush exit 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 10 client.4362.objecter successfully canceled tick 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 >> v1:172.21.15.171:6800/3417775508 conn(0x7f5eb406e110 legacy=0x7f5eb40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 10 monclient: shutdown 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 >> v1:172.21.15.87:6789/0 conn(0x7f5ed41430c0 legacy=0x7f5ed41e2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 shutdown_connections 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.417+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 >> 172.21.15.87:0/1561929905 conn(0x7f5ed40b7ea0 msgr2=0x7f5ed4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:49.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.418+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 shutdown_connections 2024-03-25T18:17:49.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.418+0000 7f5ed950e640 1 -- 172.21.15.87:0/1561929905 wait complete. 2024-03-25T18:17:49.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:49.418+0000 7f5ed950e640 1 librados: shutdown 2024-03-25T18:17:49.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:56.239 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:56.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.350+0000 7fc52b7d7640 10 monclient: get_monmap_and_config 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.350+0000 7fc52b7d7640 10 monclient: build_initial_monmap 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.350+0000 7fc52b7d7640 10 monclient: monmap: 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:17:56.351789+0000 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:17:56.351789+0000 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.352 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:56.353 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:17:56.353 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:17:56.353 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.351+0000 7fc52b7d7640 1 Processor -- start 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.352+0000 7fc52b7d7640 1 -- start start 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.352+0000 7fc52b7d7640 10 monclient: init 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.352+0000 7fc52b7d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.352+0000 7fc52b7d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-b con 0x7fc5240a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:56.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-a con 0x7fc52413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-c con 0x7fc5241430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc524142520 con 0x7fc52413f710 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b28a0 con 0x7fc5241430c0 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b2a30 con 0x7fc5240a98f0 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:56.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.353+0000 7fc52b7d7640 10 monclient(hunting): authenticate will time out at 7991.647925s 2024-03-25T18:17:56.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.361+0000 7fc528d72640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc52413f710 0x7fc5241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:59626/0 (socket says 172.21.15.87:59626) 2024-03-25T18:17:56.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.361+0000 7fc528d72640 1 -- 172.21.15.87:0/3315965552 learned_addr learned my addr 172.21.15.87:0/3315965552 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428849653 0 0) 0x7fc5240b28a0 con 0x7fc5241430c0 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:17:56.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.378+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc510003520 con 0x7fc5241430c0 2024-03-25T18:17:56.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443919024 0 0) 0x7fc5240b2a30 con 0x7fc5240a98f0 2024-03-25T18:17:56.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:17:56.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.383+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5240b28a0 con 0x7fc5240a98f0 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692991199 0 0) 0x7fc524142520 con 0x7fc52413f710 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:17:56.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.388+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5240b2a30 con 0x7fc52413f710 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181958932 0 0) 0x7fc510003520 con 0x7fc5241430c0 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 >> v1:172.21.15.87:6790/0 conn(0x7fc5240a98f0 legacy=0x7fc5240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 >> v1:172.21.15.87:6789/0 conn(0x7fc52413f710 legacy=0x7fc5241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 monclient: found mon.noname-c 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:56.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240b44c0 con 0x7fc5241430c0 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc51c001a10 con 0x7fc5241430c0 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.395+0000 7fc51bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.396+0000 7fc51bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.396+0000 7fc51bfff640 10 monclient: dump: 2024-03-25T18:17:56.398 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.396+0000 7fc52b7d7640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:17:56.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.396+0000 7fc52b7d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:56.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc51c001b70 con 0x7fc5241430c0 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 1 -- 172.21.15.87:0/3315965552 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc51c001f80 con 0x7fc5241430c0 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc52b7d7640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc52b7d7640 4 set_mon_vals no callback set 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.418+0000 7fc51bfff640 10 monclient: dump: 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.420 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.421 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.419+0000 7fc52b7d7640 10 monclient: shutdown 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.419+0000 7fc52b7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.419+0000 7fc52b7d7640 1 -- 172.21.15.87:0/3315965552 >> v1:172.21.15.171:6789/0 conn(0x7fc5241430c0 legacy=0x7fc52414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.419+0000 7fc52b7d7640 1 -- 172.21.15.87:0/3315965552 shutdown_connections 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.419+0000 7fc52b7d7640 1 -- 172.21.15.87:0/3315965552 >> 172.21.15.87:0/3315965552 conn(0x7fc5240b7ea0 msgr2=0x7fc5240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.420+0000 7fc52b7d7640 1 -- 172.21.15.87:0/3315965552 shutdown_connections 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.420+0000 7fc52b7d7640 1 -- 172.21.15.87:0/3315965552 wait complete. 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.420+0000 7fc52b7d7640 10 monclient: build_initial_monmap 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.420+0000 7fc52b7d7640 10 monclient: monmap: 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:56.424 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 librados: starting msgr at 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 librados: starting objecter 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 Processor -- start 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 -- start start 2024-03-25T18:17:56.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 librados: setting wanted keys 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 1 librados: calling monclient init 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.421+0000 7fc52b7d7640 10 monclient: init 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-c con 0x7fc5240a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-a con 0x7fc52413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient(hunting): picked mon.noname-b con 0x7fc5241430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5241db170 con 0x7fc52413f710 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5241dac50 con 0x7fc5241430c0 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5241dade0 con 0x7fc5240a98f0 2024-03-25T18:17:56.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:56.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.422+0000 7fc52b7d7640 10 monclient(hunting): authenticate will time out at 7991.717299s 2024-03-25T18:17:56.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.432+0000 7fc529573640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc5240a98f0 0x7fc5241394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:42344/0 (socket says 172.21.15.87:42344) 2024-03-25T18:17:56.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.432+0000 7fc529573640 1 -- 172.21.15.87:0/2613285848 learned_addr learned my addr 172.21.15.87:0/2613285848 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:17:56.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.433+0000 7fc519ffb640 10 client.?.objecter ms_handle_connect 0x7fc5240a98f0 2024-03-25T18:17:56.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.433+0000 7fc519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.435+0000 7fc519ffb640 10 client.?.objecter ms_handle_connect 0x7fc5241430c0 2024-03-25T18:17:56.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.435+0000 7fc519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.436+0000 7fc519ffb640 10 client.?.objecter ms_handle_connect 0x7fc52413f710 2024-03-25T18:17:56.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.436+0000 7fc519ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474571879 0 0) 0x7fc5241db170 con 0x7fc52413f710 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:17:56.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.445+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc508004eb0 con 0x7fc52413f710 2024-03-25T18:17:56.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950116877 0 0) 0x7fc5241dade0 con 0x7fc5240a98f0 2024-03-25T18:17:56.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:17:56.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.452+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5241db170 con 0x7fc5240a98f0 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292071499 0 0) 0x7fc5241dac50 con 0x7fc5241430c0 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:17:56.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.455+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5241dade0 con 0x7fc5241430c0 2024-03-25T18:17:56.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108139534 0 0) 0x7fc508004eb0 con 0x7fc52413f710 2024-03-25T18:17:56.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5241dac50 con 0x7fc52413f710 2024-03-25T18:17:56.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5140018f0 con 0x7fc52413f710 2024-03-25T18:17:56.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.465 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.466 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.466 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.463+0000 7fc519ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:56.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 343238667 0 0) 0x7fc5241db170 con 0x7fc5240a98f0 2024-03-25T18:17:56.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc508004eb0 con 0x7fc5240a98f0 2024-03-25T18:17:56.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5200019b0 con 0x7fc5240a98f0 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.474+0000 7fc519ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.476 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41248012 0 0) 0x7fc5241dade0 con 0x7fc5241430c0 2024-03-25T18:17:56.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5241db170 con 0x7fc5241430c0 2024-03-25T18:17:56.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc51c001e80 con 0x7fc5241430c0 2024-03-25T18:17:56.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.476+0000 7fc519ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.478 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116753235 0 0) 0x7fc5241dac50 con 0x7fc52413f710 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 >> v1:172.21.15.87:6790/0 conn(0x7fc5240a98f0 legacy=0x7fc5241394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 >> v1:172.21.15.171:6789/0 conn(0x7fc5241430c0 legacy=0x7fc524132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 1 monclient: found mon.a 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52413c170 con 0x7fc52413f710 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:56.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 10 monclient: _renew_subs 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52413c420 con 0x7fc52413f710 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 10 monclient: _renew_subs 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524133860 con 0x7fc52413f710 2024-03-25T18:17:56.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.490+0000 7fc52b7d7640 1 librados: init done 2024-03-25T18:17:56.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.491+0000 7fc52b7d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:56.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.491+0000 7fc52b7d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:56.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.491+0000 7fc52b7d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:56.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.491+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5241daf70 con 0x7fc52413f710 2024-03-25T18:17:56.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc514001d40 con 0x7fc52413f710 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc514004a80 con 0x7fc52413f710 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc519ffb640 10 monclient: dump: 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.815 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:17:56.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.813+0000 7fc51b7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:56.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.814+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fc51401ab10 con 0x7fc52413f710 2024-03-25T18:17:56.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.815+0000 7fc529573640 10 monclient: get_auth_request con 0x7fc50806e0b0 auth_method 0 2024-03-25T18:17:56.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.815+0000 7fc519ffb640 10 client.4368.objecter ms_handle_connect 0x7fc50806e0b0 2024-03-25T18:17:56.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fc514054740 con 0x7fc52413f710 2024-03-25T18:17:56.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 10 client.4368.objecter ms_dispatch 0x7fc524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:56.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:56.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:56.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:17:56.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc514050250 con 0x7fc52413f710 2024-03-25T18:17:56.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:56.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:56.822+0000 7fc519ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:57.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.048+0000 7fc52b7d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.048+0000 7fc52b7d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.048+0000 7fc52b7d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:17:57.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.048+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5240627b0 con 0x7fc52413f710 2024-03-25T18:17:57.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.079+0000 7fc519ffb640 1 -- 172.21.15.87:0/2613285848 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fc514052180 con 0x7fc52413f710 2024-03-25T18:17:57.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.079+0000 7fc519ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:57.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.079+0000 7fc519ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:57.081 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:17:57.081 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:57.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 10 librados: watch_flush enter 2024-03-25T18:17:57.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 10 librados: watch_flush exit 2024-03-25T18:17:57.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:17:57.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 >> v1:172.21.15.171:6800/3417775508 conn(0x7fc50806e0b0 legacy=0x7fc508070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 10 monclient: shutdown 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 >> v1:172.21.15.87:6789/0 conn(0x7fc52413f710 legacy=0x7fc5241d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 shutdown_connections 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 >> 172.21.15.87:0/2613285848 conn(0x7fc5240b7ea0 msgr2=0x7fc52414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 shutdown_connections 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 -- 172.21.15.87:0/2613285848 wait complete. 2024-03-25T18:17:57.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:17:57.082+0000 7fc52b7d7640 1 librados: shutdown 2024-03-25T18:17:57.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:01.926 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:49.639 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.937 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:49.650 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.971 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:03:49.684 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.988 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:49.667 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.989 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:49.701 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.993 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:49.672 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:01.998 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:49.711 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:02.029 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:03:49.742 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:02.037 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:03:49.716 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.901 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:04.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.011+0000 7f5c30dd6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.011+0000 7f5c30dd6640 10 monclient: build_initial_monmap 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.011+0000 7f5c30dd6640 10 monclient: monmap: 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:04.012976+0000 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:04.012976+0000 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:04.013 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.012+0000 7f5c30dd6640 1 Processor -- start 2024-03-25T18:18:04.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.012+0000 7f5c30dd6640 1 -- start start 2024-03-25T18:18:04.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.012+0000 7f5c30dd6640 10 monclient: init 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c0705b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c1402d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0718c0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c0714c0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c072b50 con 0x7f5c2c1402d0 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:04.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.013+0000 7f5c30dd6640 10 monclient(hunting): authenticate will time out at 7999.309184s 2024-03-25T18:18:04.021 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.019+0000 7f5c29d9b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f5c2c1402d0 0x7f5c2c140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:43364/0 (socket says 172.21.15.87:43364) 2024-03-25T18:18:04.021 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.019+0000 7f5c29d9b640 1 -- 172.21.15.87:0/1825568129 learned_addr learned my addr 172.21.15.87:0/1825568129 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793412506 0 0) 0x7f5c2c072b50 con 0x7f5c2c1402d0 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:18:04.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.036+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c14003520 con 0x7f5c2c1402d0 2024-03-25T18:18:04.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384060046 0 0) 0x7f5c2c0718c0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:18:04.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.042+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c072b50 con 0x7f5c2c0705b0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265302760 0 0) 0x7f5c14003520 con 0x7f5c2c1402d0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 >> v1:172.21.15.171:6789/0 conn(0x7f5c2c149ae0 legacy=0x7f5c2c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 >> v1:172.21.15.87:6789/0 conn(0x7f5c2c0705b0 legacy=0x7f5c2c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 monclient: found mon.noname-b 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c13dd10 con 0x7f5c2c1402d0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: _finish_auth 0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c20001690 con 0x7f5c2c1402d0 2024-03-25T18:18:04.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c2959a640 10 monclient: dump: 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c30dd6640 5 monclient: authenticate success, global_id 4379 2024-03-25T18:18:04.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.048+0000 7f5c30dd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:04.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c200017f0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 1 -- 172.21.15.87:0/1825568129 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c20001c00 con 0x7f5c2c1402d0 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c2959a640 10 monclient: dump: 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.057 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c30dd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:04.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.054+0000 7f5c30dd6640 4 set_mon_vals no callback set 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.055+0000 7f5c30dd6640 10 monclient: shutdown 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.055+0000 7f5c30dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.055+0000 7f5c30dd6640 1 -- 172.21.15.87:0/1825568129 >> v1:172.21.15.87:6790/0 conn(0x7f5c2c1402d0 legacy=0x7f5c2c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.055+0000 7f5c30dd6640 1 -- 172.21.15.87:0/1825568129 shutdown_connections 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.055+0000 7f5c30dd6640 1 -- 172.21.15.87:0/1825568129 >> 172.21.15.87:0/1825568129 conn(0x7f5c2c0b5d10 msgr2=0x7f5c2c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.056+0000 7f5c30dd6640 1 -- 172.21.15.87:0/1825568129 shutdown_connections 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.056+0000 7f5c30dd6640 1 -- 172.21.15.87:0/1825568129 wait complete. 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.056+0000 7f5c30dd6640 10 monclient: build_initial_monmap 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.056+0000 7f5c30dd6640 10 monclient: monmap: 2024-03-25T18:18:04.059 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 librados: starting msgr at 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 librados: starting objecter 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 Processor -- start 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 -- start start 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 librados: setting wanted keys 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 1 librados: calling monclient init 2024-03-25T18:18:04.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.057+0000 7f5c30dd6640 10 monclient: init 2024-03-25T18:18:04.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:04.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:04.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c0705b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c1402d0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c149ae0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c1470d0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c1dba00 con 0x7f5c2c1402d0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c1423c0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:04.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.058+0000 7f5c30dd6640 10 monclient(hunting): authenticate will time out at 7999.354454s 2024-03-25T18:18:04.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.074+0000 7f5c29d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5c2c1402d0 0x7f5c2c1dbf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:45938/0 (socket says 172.21.15.87:45938) 2024-03-25T18:18:04.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.074+0000 7f5c29d9b640 1 -- 172.21.15.87:0/2174953681 learned_addr learned my addr 172.21.15.87:0/2174953681 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:04.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.075+0000 7f5c1affd640 10 client.?.objecter ms_handle_connect 0x7f5c2c1402d0 2024-03-25T18:18:04.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.075+0000 7f5c1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.088+0000 7f5c1affd640 10 client.?.objecter ms_handle_connect 0x7f5c2c0705b0 2024-03-25T18:18:04.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.088+0000 7f5c1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.091+0000 7f5c1affd640 10 client.?.objecter ms_handle_connect 0x7f5c2c149ae0 2024-03-25T18:18:04.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.091+0000 7f5c1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318784016 0 0) 0x7f5c2c1470d0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:18:04.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.099+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c004eb0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.102+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819297290 0 0) 0x7f5c2c1423c0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.102+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.102+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.102+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.102+0000 7f5c1affd640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:18:04.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.103+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c1470d0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127485647 0 0) 0x7f5c2c1dba00 con 0x7f5c2c1402d0 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:18:04.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.105+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c1423c0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730592772 0 0) 0x7f5c2c1423c0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c2c1dba00 con 0x7f5c2c1402d0 2024-03-25T18:18:04.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c20001bc0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 10 monclient(hunting): dump: 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.121+0000 7f5c1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910730567 0 0) 0x7f5c2c1470d0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c2c1423c0 con 0x7f5c2c149ae0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3251219393 0 0) 0x7f5c0c004eb0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c2c1470d0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c24001b00 con 0x7f5c2c149ae0 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): dump: 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c1c0018b0 con 0x7f5c2c0705b0 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.123+0000 7f5c1affd640 10 monclient(hunting): dump: 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.127 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2688452832 0 0) 0x7f5c2c1dba00 con 0x7f5c2c1402d0 2024-03-25T18:18:04.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 >> v1:172.21.15.87:6790/0 conn(0x7f5c2c149ae0 legacy=0x7f5c2c148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 >> v1:172.21.15.87:6789/0 conn(0x7f5c2c0705b0 legacy=0x7f5c2c1459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:04.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 1 monclient: found mon.b 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c142550 con 0x7f5c2c1402d0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 10 monclient: _renew_subs 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c144e10 con 0x7f5c2c1402d0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 10 monclient: _renew_subs 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c1462b0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.139+0000 7f5c30dd6640 1 librados: init done 2024-03-25T18:18:04.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.141+0000 7f5c30dd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.141+0000 7f5c30dd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.141+0000 7f5c30dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:04.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.141+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c0627b0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c20004a90 con 0x7f5c2c1402d0 2024-03-25T18:18:04.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:04.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5c20004da0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:04.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c1affd640 10 monclient: dump: 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:04.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.431+0000 7f5c28d99640 4 set_mon_vals no callback set 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.432+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f5c2001ad70 con 0x7f5c2c1402d0 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.433+0000 7f5c2a59c640 10 monclient: get_auth_request con 0x7f5c0c06e1b0 auth_method 0 2024-03-25T18:18:04.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.433+0000 7f5c1affd640 10 client.4381.objecter ms_handle_connect 0x7f5c0c06e1b0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.436+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f5c200548f0 con 0x7f5c2c1402d0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.436+0000 7f5c1affd640 10 client.4381.objecter ms_dispatch 0x7f5c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.436+0000 7f5c1affd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:04.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.436+0000 7f5c1affd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:04.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.436+0000 7f5c1affd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:18:04.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.437+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c20050400 con 0x7f5c2c1402d0 2024-03-25T18:18:04.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.437+0000 7f5c1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:04.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.437+0000 7f5c1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:04.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.644+0000 7f5c30dd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:04.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.644+0000 7f5c30dd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:04.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.644+0000 7f5c30dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:04.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.644+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c2c1dbb90 con 0x7f5c2c1402d0 2024-03-25T18:18:04.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.774+0000 7f5c1affd640 1 -- 172.21.15.87:0/2174953681 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f5c20052330 con 0x7f5c2c1402d0 2024-03-25T18:18:04.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.774+0000 7f5c1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:04.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.774+0000 7f5c1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:04.777 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:04.777 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 10 librados: watch_flush enter 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 10 librados: watch_flush exit 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 >> v1:172.21.15.171:6800/3417775508 conn(0x7f5c0c06e1b0 legacy=0x7f5c0c070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 10 monclient: shutdown 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 >> v1:172.21.15.171:6789/0 conn(0x7f5c2c1402d0 legacy=0x7f5c2c1dbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:04.779 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 shutdown_connections 2024-03-25T18:18:04.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 >> 172.21.15.87:0/2174953681 conn(0x7f5c2c0b5d10 msgr2=0x7f5c2c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:04.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 shutdown_connections 2024-03-25T18:18:04.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 -- 172.21.15.87:0/2174953681 wait complete. 2024-03-25T18:18:04.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:04.777+0000 7f5c30dd6640 1 librados: shutdown 2024-03-25T18:18:04.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:11.599 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:11.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 10 monclient: get_monmap_and_config 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 10 monclient: build_initial_monmap 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 10 monclient: monmap: 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:11.714919+0000 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:11.714919+0000 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:11.715 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 1 Processor -- start 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 1 -- start start 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.714+0000 7f84eabc4640 10 monclient: init 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-a con 0x7f84e40a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-b con 0x7f84e413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-c con 0x7f84e41430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e4142520 con 0x7f84e40a98f0 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40b28a0 con 0x7f84e41430c0 2024-03-25T18:18:11.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40b2a30 con 0x7f84e413f710 2024-03-25T18:18:11.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:11.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.715+0000 7f84eabc4640 10 monclient(hunting): authenticate will time out at 8007.010708s 2024-03-25T18:18:11.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.732+0000 7f84e8960640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f84e40a98f0 0x7f84e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:41876/0 (socket says 172.21.15.87:41876) 2024-03-25T18:18:11.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.732+0000 7f84e8960640 1 -- 172.21.15.87:0/2948563924 learned_addr learned my addr 172.21.15.87:0/2948563924 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:11.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182157301 0 0) 0x7f84e4142520 con 0x7f84e40a98f0 2024-03-25T18:18:11.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:11.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:11.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:11.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:18:11.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.746+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d4003520 con 0x7f84e40a98f0 2024-03-25T18:18:11.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489747843 0 0) 0x7f84e40b28a0 con 0x7f84e41430c0 2024-03-25T18:18:11.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:11.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:11.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:11.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:18:11.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.749+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e4142520 con 0x7f84e41430c0 2024-03-25T18:18:11.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504669043 0 0) 0x7f84d4003520 con 0x7f84e40a98f0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 >> v1:172.21.15.87:6790/0 conn(0x7f84e413f710 legacy=0x7f84e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 >> v1:172.21.15.171:6789/0 conn(0x7f84e41430c0 legacy=0x7f84e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 monclient: found mon.noname-a 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40b44c0 con 0x7f84e40a98f0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f84d8001690 con 0x7f84e40a98f0 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:11.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84e37fe640 10 monclient: dump: 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84eabc4640 5 monclient: authenticate success, global_id 4377 2024-03-25T18:18:11.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.754+0000 7f84eabc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d80017f0 con 0x7f84e40a98f0 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 1 -- 172.21.15.87:0/2948563924 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f84d8001c00 con 0x7f84e40a98f0 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:11.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84e37fe640 10 monclient: dump: 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84eabc4640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:11.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.757+0000 7f84eabc4640 4 set_mon_vals no callback set 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 10 monclient: shutdown 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 1 -- 172.21.15.87:0/2948563924 >> v1:172.21.15.87:6789/0 conn(0x7f84e40a98f0 legacy=0x7f84e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 1 -- 172.21.15.87:0/2948563924 shutdown_connections 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 1 -- 172.21.15.87:0/2948563924 >> 172.21.15.87:0/2948563924 conn(0x7f84e40b7ea0 msgr2=0x7f84e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 1 -- 172.21.15.87:0/2948563924 shutdown_connections 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.758+0000 7f84eabc4640 1 -- 172.21.15.87:0/2948563924 wait complete. 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.759+0000 7f84eabc4640 10 monclient: build_initial_monmap 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.759+0000 7f84eabc4640 10 monclient: monmap: 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:11.761 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.759+0000 7f84eabc4640 1 librados: starting msgr at 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.759+0000 7f84eabc4640 1 librados: starting objecter 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 1 Processor -- start 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 1 -- start start 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 1 librados: setting wanted keys 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 1 librados: calling monclient init 2024-03-25T18:18:11.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 10 monclient: init 2024-03-25T18:18:11.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-a con 0x7f84e40a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.760+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-b con 0x7f84e413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 10 monclient(hunting): picked mon.noname-c con 0x7f84e41430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e41369b0 con 0x7f84e40a98f0 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e41366b0 con 0x7f84e413f710 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e4133ad0 con 0x7f84e41430c0 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:11.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.761+0000 7f84eabc4640 10 monclient(hunting): authenticate will time out at 8007.056031s 2024-03-25T18:18:11.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.773+0000 7f84e9161640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f84e41430c0 0x7f84e41333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:51506/0 (socket says 172.21.15.87:51506) 2024-03-25T18:18:11.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.773+0000 7f84e9161640 1 -- 172.21.15.87:0/2733831580 learned_addr learned my addr 172.21.15.87:0/2733831580 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:11.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.774+0000 7f84e17fa640 10 client.?.objecter ms_handle_connect 0x7f84e41430c0 2024-03-25T18:18:11.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.774+0000 7f84e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:11.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.775+0000 7f84e17fa640 10 client.?.objecter ms_handle_connect 0x7f84e413f710 2024-03-25T18:18:11.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.775+0000 7f84e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:11.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.782+0000 7f84e17fa640 10 client.?.objecter ms_handle_connect 0x7f84e40a98f0 2024-03-25T18:18:11.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.782+0000 7f84e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:11.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.792+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670800009 0 0) 0x7f84e41366b0 con 0x7f84e413f710 2024-03-25T18:18:11.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:11.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:11.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:11.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:18:11.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c8004eb0 con 0x7f84e413f710 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662145377 0 0) 0x7f84e41369b0 con 0x7f84e40a98f0 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:18:11.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.793+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e41366b0 con 0x7f84e40a98f0 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783044803 0 0) 0x7f84e4133ad0 con 0x7f84e41430c0 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:18:11.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.794+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e41369b0 con 0x7f84e41430c0 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4109784572 0 0) 0x7f84c8004eb0 con 0x7f84e413f710 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e4133ad0 con 0x7f84e413f710 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f84d00018c0 con 0x7f84e413f710 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 10 monclient(hunting): dump: 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:11.816 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:11.817 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:11.817 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:11.817 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:11.817 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.815+0000 7f84e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696200935 0 0) 0x7f84e41369b0 con 0x7f84e41430c0 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84c8004eb0 con 0x7f84e41430c0 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f84dc001bd0 con 0x7f84e41430c0 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.823+0000 7f84e17fa640 10 monclient(hunting): dump: 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:11.824 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:11.825 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:11.825 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:11.825 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:11.825 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:11.825 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.825+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3957304086 0 0) 0x7f84e4133ad0 con 0x7f84e413f710 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 >> v1:172.21.15.87:6790/0 conn(0x7f84e41430c0 legacy=0x7f84e41333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 >> v1:172.21.15.87:6789/0 conn(0x7f84e40a98f0 legacy=0x7f84e41371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 1 monclient: found mon.b 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4136e40 con 0x7f84e413f710 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:11.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 10 monclient: _renew_subs 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e41378f0 con 0x7f84e413f710 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 10 monclient: _renew_subs 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e40b1480 con 0x7f84e413f710 2024-03-25T18:18:11.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.826+0000 7f84eabc4640 1 librados: init done 2024-03-25T18:18:11.829 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.827+0000 7f84eabc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:11.829 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.827+0000 7f84eabc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:11.829 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.827+0000 7f84eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:11.829 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:11.827+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4137030 con 0x7f84e413f710 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d00035f0 con 0x7f84e413f710 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f84d0003a00 con 0x7f84e413f710 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e17fa640 10 monclient: dump: 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:12.113 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.112+0000 7f84e2ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:12.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.113+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f84d0003c20 con 0x7f84e413f710 2024-03-25T18:18:12.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.113+0000 7f84e8960640 10 monclient: get_auth_request con 0x7f84c806df70 auth_method 0 2024-03-25T18:18:12.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.114+0000 7f84e17fa640 10 client.4387.objecter ms_handle_connect 0x7f84c806df70 2024-03-25T18:18:12.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f84d0053590 con 0x7f84e413f710 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 10 client.4387.objecter ms_dispatch 0x7f84e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84d004f0a0 con 0x7f84e413f710 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:12.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.119+0000 7f84e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:12.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.337+0000 7f84eabc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.337+0000 7f84eabc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.337+0000 7f84eabc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:12.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.337+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84e4136b60 con 0x7f84e413f710 2024-03-25T18:18:12.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.381+0000 7f84e17fa640 1 -- 172.21.15.87:0/2733831580 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f84e4136b60 con 0x7f84e413f710 2024-03-25T18:18:12.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.382+0000 7f84e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:12.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.382+0000 7f84e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:12.383 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:12.383 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 10 librados: watch_flush enter 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 10 librados: watch_flush exit 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 10 client.4387.objecter successfully canceled tick 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 >> v1:172.21.15.171:6800/3417775508 conn(0x7f84c806df70 legacy=0x7f84c8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 10 monclient: shutdown 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 >> v1:172.21.15.171:6789/0 conn(0x7f84e413f710 legacy=0x7f84e4132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 shutdown_connections 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 >> 172.21.15.87:0/2733831580 conn(0x7f84e40b7ea0 msgr2=0x7f84e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 shutdown_connections 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.384+0000 7f84eabc4640 1 -- 172.21.15.87:0/2733831580 wait complete. 2024-03-25T18:18:12.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:12.385+0000 7f84eabc4640 1 librados: shutdown 2024-03-25T18:18:12.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:19.203 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.315+0000 7fb13c281640 10 monclient: get_monmap_and_config 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.315+0000 7fb13c281640 10 monclient: build_initial_monmap 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.315+0000 7fb13c281640 10 monclient: monmap: 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:19.316193+0000 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:19.316193+0000 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:19.316 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.317 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:19.317 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:19.317 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:19.317 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.316+0000 7fb13c281640 1 Processor -- start 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.316+0000 7fb13c281640 1 -- start start 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.316+0000 7fb13c281640 10 monclient: init 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.316+0000 7fb13c281640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-a con 0x7fb1340a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-b con 0x7fb13413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-c con 0x7fb1341430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134142520 con 0x7fb1340a98f0 2024-03-25T18:18:19.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340b28a0 con 0x7fb1341430c0 2024-03-25T18:18:19.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340b2a30 con 0x7fb13413f710 2024-03-25T18:18:19.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:19.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.317+0000 7fb13c281640 10 monclient(hunting): authenticate will time out at 8014.612098s 2024-03-25T18:18:19.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.319+0000 7fb13981c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fb13413f710 0x7fb1341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:38416/0 (socket says 172.21.15.87:38416) 2024-03-25T18:18:19.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.319+0000 7fb13981c640 1 -- 172.21.15.87:0/2626497944 learned_addr learned my addr 172.21.15.87:0/2626497944 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888662602 0 0) 0x7fb134142520 con 0x7fb1340a98f0 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:18:19.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.328+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120003520 con 0x7fb1340a98f0 2024-03-25T18:18:19.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.331+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537078138 0 0) 0x7fb1340b2a30 con 0x7fb13413f710 2024-03-25T18:18:19.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.332+0000 7fb13901b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.332+0000 7fb13901b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.332+0000 7fb13901b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.332+0000 7fb13901b640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:18:19.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.332+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134142520 con 0x7fb13413f710 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359723764 0 0) 0x7fb120003520 con 0x7fb1340a98f0 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 >> v1:172.21.15.87:6790/0 conn(0x7fb13413f710 legacy=0x7fb1341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 >> v1:172.21.15.171:6789/0 conn(0x7fb1341430c0 legacy=0x7fb13414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 monclient: found mon.noname-a 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340b44c0 con 0x7fb1340a98f0 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: _finish_auth 0 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:19.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb130001690 con 0x7fb1340a98f0 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13901b640 10 monclient: dump: 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13c281640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:18:19.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.339+0000 7fb13c281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:19.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1300017f0 con 0x7fb1340a98f0 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 1 -- 172.21.15.87:0/2626497944 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb130001c00 con 0x7fb1340a98f0 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13c281640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13c281640 4 set_mon_vals no callback set 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.346+0000 7fb13901b640 10 monclient: dump: 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.348 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.349 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.349 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.349 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.349 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.347+0000 7fb13c281640 10 monclient: shutdown 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.347+0000 7fb13c281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.347+0000 7fb13c281640 1 -- 172.21.15.87:0/2626497944 >> v1:172.21.15.87:6789/0 conn(0x7fb1340a98f0 legacy=0x7fb1340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 1 -- 172.21.15.87:0/2626497944 shutdown_connections 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 1 -- 172.21.15.87:0/2626497944 >> 172.21.15.87:0/2626497944 conn(0x7fb1340b7ea0 msgr2=0x7fb1340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 1 -- 172.21.15.87:0/2626497944 shutdown_connections 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 1 -- 172.21.15.87:0/2626497944 wait complete. 2024-03-25T18:18:19.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 10 monclient: build_initial_monmap 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.348+0000 7fb13c281640 10 monclient: monmap: 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 librados: starting msgr at 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 librados: starting objecter 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 Processor -- start 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 -- start start 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 librados: setting wanted keys 2024-03-25T18:18:19.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 1 librados: calling monclient init 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.349+0000 7fb13c281640 10 monclient: init 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-a con 0x7fb1340a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-b con 0x7fb13413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient(hunting): picked mon.noname-c con 0x7fb1341430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134136b30 con 0x7fb1340a98f0 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134137710 con 0x7fb13413f710 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134137280 con 0x7fb1341430c0 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:19.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.350+0000 7fb13c281640 10 monclient(hunting): authenticate will time out at 8014.645037s 2024-03-25T18:18:19.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.353+0000 7fb13981c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb13413f710 0x7fb1341d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:38720/0 (socket says 172.21.15.87:38720) 2024-03-25T18:18:19.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.353+0000 7fb13981c640 1 -- 172.21.15.87:0/3622977584 learned_addr learned my addr 172.21.15.87:0/3622977584 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:19.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.358+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb13413f710 2024-03-25T18:18:19.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.358+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.364 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.364+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb1340a98f0 2024-03-25T18:18:19.364 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.364+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505115505 0 0) 0x7fb134137710 con 0x7fb13413f710 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:18:19.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.368+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb118004eb0 con 0x7fb13413f710 2024-03-25T18:18:19.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793950644 0 0) 0x7fb134136b30 con 0x7fb1340a98f0 2024-03-25T18:18:19.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:18:19.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.370+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134137710 con 0x7fb1340a98f0 2024-03-25T18:18:19.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.374+0000 7fb12affd640 10 client.?.objecter ms_handle_connect 0x7fb1341430c0 2024-03-25T18:18:19.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.374+0000 7fb12affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345699268 0 0) 0x7fb134137280 con 0x7fb1341430c0 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:18:19.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.383+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134136b30 con 0x7fb1341430c0 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495902403 0 0) 0x7fb118004eb0 con 0x7fb13413f710 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb134137280 con 0x7fb13413f710 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb1240019f0 con 0x7fb13413f710 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): dump: 2024-03-25T18:18:19.395 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399448565 0 0) 0x7fb134136b30 con 0x7fb1341430c0 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb118004eb0 con 0x7fb1341430c0 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb12c0019d0 con 0x7fb1341430c0 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:19.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.394+0000 7fb12affd640 10 monclient(hunting): dump: 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.397 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686262420 0 0) 0x7fb134137710 con 0x7fb1340a98f0 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb134136b30 con 0x7fb1340a98f0 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb130001b40 con 0x7fb1340a98f0 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.395+0000 7fb12affd640 10 monclient(hunting): dump: 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.398 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.399 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2567404631 0 0) 0x7fb134136b30 con 0x7fb1340a98f0 2024-03-25T18:18:19.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 >> v1:172.21.15.87:6790/0 conn(0x7fb1341430c0 legacy=0x7fb134132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 >> v1:172.21.15.171:6789/0 conn(0x7fb13413f710 legacy=0x7fb1341d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 1 monclient: found mon.a 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb134137410 con 0x7fb1340a98f0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb12affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 10 monclient: _renew_subs 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13413c2e0 con 0x7fb1340a98f0 2024-03-25T18:18:19.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 10 monclient: _renew_subs 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb134133610 con 0x7fb1340a98f0 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.402+0000 7fb13c281640 1 librados: init done 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.403+0000 7fb13c281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.403+0000 7fb13c281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.403+0000 7fb13c281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.403+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1341378a0 con 0x7fb1340a98f0 2024-03-25T18:18:19.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1300038a0 con 0x7fb1340a98f0 2024-03-25T18:18:19.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:19.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb130003bb0 con 0x7fb1340a98f0 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb12affd640 10 monclient: dump: 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:19.716 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:19.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.714+0000 7fb13881a640 4 set_mon_vals no callback set 2024-03-25T18:18:19.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.715+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fb130019b80 con 0x7fb1340a98f0 2024-03-25T18:18:19.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.716+0000 7fb13981c640 10 monclient: get_auth_request con 0x7fb11806e160 auth_method 0 2024-03-25T18:18:19.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.716+0000 7fb12affd640 10 client.4386.objecter ms_handle_connect 0x7fb11806e160 2024-03-25T18:18:19.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fb1300536f0 con 0x7fb1340a98f0 2024-03-25T18:18:19.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 10 client.4386.objecter ms_dispatch 0x7fb134000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:19.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:19.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:19.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:18:19.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb13004f030 con 0x7fb1340a98f0 2024-03-25T18:18:19.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:19.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.721+0000 7fb12affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:19.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.951+0000 7fb13c281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:19.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.951+0000 7fb13c281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:19.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.951+0000 7fb13c281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:19.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.951+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1340627b0 con 0x7fb1340a98f0 2024-03-25T18:18:19.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.993+0000 7fb12affd640 1 -- 172.21.15.87:0/3622977584 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fb130051130 con 0x7fb1340a98f0 2024-03-25T18:18:19.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.993+0000 7fb12affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:19.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.993+0000 7fb12affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:19.994 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:19.994 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 10 librados: watch_flush enter 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 10 librados: watch_flush exit 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 >> v1:172.21.15.171:6800/3417775508 conn(0x7fb11806e160 legacy=0x7fb118070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 10 monclient: shutdown 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.995+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 >> v1:172.21.15.87:6789/0 conn(0x7fb1340a98f0 legacy=0x7fb134137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.996+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 shutdown_connections 2024-03-25T18:18:19.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.996+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 >> 172.21.15.87:0/3622977584 conn(0x7fb1340b7ea0 msgr2=0x7fb134074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:19.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.996+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 shutdown_connections 2024-03-25T18:18:19.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.996+0000 7fb13c281640 1 -- 172.21.15.87:0/3622977584 wait complete. 2024-03-25T18:18:19.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:19.996+0000 7fb13c281640 1 librados: shutdown 2024-03-25T18:18:20.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:22.008 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:09.721 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.019 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:09.733 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.056 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:09.769 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.074 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:09.787 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.083 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:09.796 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.085 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:09.764 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.091 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:09.769 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.111 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:09.790 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.112 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:09.824 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:26.813 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.927+0000 7f1865a8d640 10 monclient: get_monmap_and_config 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.927+0000 7f1865a8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.927+0000 7f1865a8d640 10 monclient: monmap: 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:26.928307+0000 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:26.928307+0000 2024-03-25T18:18:26.928 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:26.929 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:26.929 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:26.929 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:26.929 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:26.929 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 1 Processor -- start 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 1 -- start start 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient: init 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f18600705b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f18601402d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f1860149ae0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:26.930 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18600718c0 con 0x7f1860149ae0 2024-03-25T18:18:26.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18600714c0 con 0x7f18600705b0 2024-03-25T18:18:26.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1860072b50 con 0x7f18601402d0 2024-03-25T18:18:26.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:26.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.928+0000 7f1865a8d640 10 monclient(hunting): authenticate will time out at 8022.223740s 2024-03-25T18:18:26.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.939+0000 7f185effd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f18601402d0 0x7f1860140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:43038/0 (socket says 172.21.15.87:43038) 2024-03-25T18:18:26.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.939+0000 7f185effd640 1 -- 172.21.15.87:0/2528288030 learned_addr learned my addr 172.21.15.87:0/2528288030 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:26.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168297012 0 0) 0x7f1860072b50 con 0x7f18601402d0 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848003520 con 0x7f18601402d0 2024-03-25T18:18:26.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557884861 0 0) 0x7f18600714c0 con 0x7f18600705b0 2024-03-25T18:18:26.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:26.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:26.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:26.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:18:26.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.956+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1860072b50 con 0x7f18600705b0 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391830252 0 0) 0x7f18600718c0 con 0x7f1860149ae0 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:18:26.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.960+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18600714c0 con 0x7f1860149ae0 2024-03-25T18:18:26.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494770870 0 0) 0x7f18600714c0 con 0x7f1860149ae0 2024-03-25T18:18:26.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 >> v1:172.21.15.87:6790/0 conn(0x7f18601402d0 legacy=0x7f1860140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:26.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 >> v1:172.21.15.171:6789/0 conn(0x7f18600705b0 legacy=0x7f186013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186013dd10 con 0x7f1860149ae0 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1858001930 con 0x7f1860149ae0 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f185e7fc640 10 monclient: dump: 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:26.974 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f1865a8d640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:18:26.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.972+0000 7f1865a8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1858001a90 con 0x7f1860149ae0 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 1 -- 172.21.15.87:0/2528288030 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1858001ee0 con 0x7f1860149ae0 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f1865a8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f1865a8d640 4 set_mon_vals no callback set 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f185e7fc640 10 monclient: dump: 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:26.988 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:26.989 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f1865a8d640 10 monclient: shutdown 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f1865a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.987+0000 7f1865a8d640 1 -- 172.21.15.87:0/2528288030 >> v1:172.21.15.87:6789/0 conn(0x7f1860149ae0 legacy=0x7f186014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 1 -- 172.21.15.87:0/2528288030 shutdown_connections 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 1 -- 172.21.15.87:0/2528288030 >> 172.21.15.87:0/2528288030 conn(0x7f18600b5d10 msgr2=0x7f18600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 1 -- 172.21.15.87:0/2528288030 shutdown_connections 2024-03-25T18:18:26.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 1 -- 172.21.15.87:0/2528288030 wait complete. 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.988+0000 7f1865a8d640 10 monclient: monmap: 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 librados: starting msgr at 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 librados: starting objecter 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 Processor -- start 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 -- start start 2024-03-25T18:18:26.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 librados: setting wanted keys 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 1 librados: calling monclient init 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.989+0000 7f1865a8d640 10 monclient: init 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f18600705b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f18601402d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f1860149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186013dfd0 con 0x7f18601402d0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1860142830 con 0x7f1860149ae0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1860141a40 con 0x7f18600705b0 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:26.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.990+0000 7f1865a8d640 10 monclient(hunting): authenticate will time out at 8022.285071s 2024-03-25T18:18:26.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.997+0000 7f185ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1860149ae0 0x7f18601e0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:43338/0 (socket says 172.21.15.87:43338) 2024-03-25T18:18:26.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:26.997+0000 7f185ffff640 1 -- 172.21.15.87:0/998909020 learned_addr learned my addr 172.21.15.87:0/998909020 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:27.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.002+0000 7f1847fff640 10 client.?.objecter ms_handle_connect 0x7f1860149ae0 2024-03-25T18:18:27.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.002+0000 7f1847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.008+0000 7f1847fff640 10 client.?.objecter ms_handle_connect 0x7f18600705b0 2024-03-25T18:18:27.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.008+0000 7f1847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.009+0000 7f1847fff640 10 client.?.objecter ms_handle_connect 0x7f18601402d0 2024-03-25T18:18:27.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.009+0000 7f1847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681193399 0 0) 0x7f1860142830 con 0x7f1860149ae0 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:18:27.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.016+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c004eb0 con 0x7f1860149ae0 2024-03-25T18:18:27.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.026+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611091021 0 0) 0x7f184c004eb0 con 0x7f1860149ae0 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1860142830 con 0x7f1860149ae0 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1858001e60 con 0x7f1860149ae0 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 10 monclient(hunting): dump: 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:27.028 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:27.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.027+0000 7f1847fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:27.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028169838 0 0) 0x7f186013dfd0 con 0x7f18601402d0 2024-03-25T18:18:27.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:18:27.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.029+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c004eb0 con 0x7f18601402d0 2024-03-25T18:18:27.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779067619 0 0) 0x7f1860141a40 con 0x7f18600705b0 2024-03-25T18:18:27.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:27.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:27.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:27.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:18:27.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.030+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186013dfd0 con 0x7f18600705b0 2024-03-25T18:18:27.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 737438578 0 0) 0x7f1860142830 con 0x7f1860149ae0 2024-03-25T18:18:27.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 >> v1:172.21.15.87:6790/0 conn(0x7f18600705b0 legacy=0x7f18601d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 >> v1:172.21.15.87:6789/0 conn(0x7f18601402d0 legacy=0x7f1860133e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 1 monclient: found mon.b 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860141bd0 con 0x7f1860149ae0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 10 monclient: _renew_subs 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860142b80 con 0x7f1860149ae0 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 10 monclient: _renew_subs 2024-03-25T18:18:27.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18601d5fa0 con 0x7f1860149ae0 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.037+0000 7f1865a8d640 1 librados: init done 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.038+0000 7f1865a8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.038+0000 7f1865a8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.038+0000 7f1865a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:27.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.038+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860142f20 con 0x7f1860149ae0 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18580020e0 con 0x7f1860149ae0 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1858003dc0 con 0x7f1860149ae0 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:27.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f1847fff640 10 monclient: dump: 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:27.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.318+0000 7f185dffb640 4 set_mon_vals no callback set 2024-03-25T18:18:27.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.319+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f1858003fe0 con 0x7f1860149ae0 2024-03-25T18:18:27.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.319+0000 7f185f7fe640 10 monclient: get_auth_request con 0x7f184c06de10 auth_method 0 2024-03-25T18:18:27.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.320+0000 7f1847fff640 10 client.4396.objecter ms_handle_connect 0x7f184c06de10 2024-03-25T18:18:27.324 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f1858054af0 con 0x7f1860149ae0 2024-03-25T18:18:27.324 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 10 client.4396.objecter ms_dispatch 0x7f1860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:27.324 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:27.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:27.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:18:27.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.323+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1858050600 con 0x7f1860149ae0 2024-03-25T18:18:27.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.324+0000 7f1847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:27.325 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.324+0000 7f1847fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:27.577 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.576+0000 7f1865a8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:27.577 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.576+0000 7f1865a8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:27.577 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.576+0000 7f1865a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:27.577 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.576+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18600627b0 con 0x7f1860149ae0 2024-03-25T18:18:27.597 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.597+0000 7f1847fff640 1 -- 172.21.15.87:0/998909020 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f1858052530 con 0x7f1860149ae0 2024-03-25T18:18:27.597 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.597+0000 7f1847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:27.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.597+0000 7f1847fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:27.598 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:27.598 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:27.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 10 librados: watch_flush enter 2024-03-25T18:18:27.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 10 librados: watch_flush exit 2024-03-25T18:18:27.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:18:27.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 >> v1:172.21.15.171:6800/3417775508 conn(0x7f184c06de10 legacy=0x7f184c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 10 monclient: shutdown 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 >> v1:172.21.15.171:6789/0 conn(0x7f1860149ae0 legacy=0x7f18601e0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 shutdown_connections 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 >> 172.21.15.87:0/998909020 conn(0x7f18600b5d10 msgr2=0x7f186013b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 shutdown_connections 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 -- 172.21.15.87:0/998909020 wait complete. 2024-03-25T18:18:27.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:27.599+0000 7f1865a8d640 1 librados: shutdown 2024-03-25T18:18:27.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:34.418 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:34.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.533+0000 7fe543443640 10 monclient: get_monmap_and_config 2024-03-25T18:18:34.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.533+0000 7fe543443640 10 monclient: build_initial_monmap 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.533+0000 7fe543443640 10 monclient: monmap: 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:34.533924+0000 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:34.533924+0000 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:34.534 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.533+0000 7fe543443640 1 Processor -- start 2024-03-25T18:18:34.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.534+0000 7fe543443640 1 -- start start 2024-03-25T18:18:34.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.534+0000 7fe543443640 10 monclient: init 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.534+0000 7fe543443640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.534+0000 7fe543443640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0af940 con 0x7fe53c0b4d60 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0b0a70 con 0x7fe53c14be90 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0b0d50 con 0x7fe53c0b1730 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:34.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.535+0000 7fe543443640 10 monclient(hunting): authenticate will time out at 8029.829916s 2024-03-25T18:18:34.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.542+0000 7fe5411df640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fe53c0b4d60 0x7fe53c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:35958/0 (socket says 172.21.15.87:35958) 2024-03-25T18:18:34.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.542+0000 7fe5411df640 1 -- 172.21.15.87:0/237526647 learned_addr learned my addr 172.21.15.87:0/237526647 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:34.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.556+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46233762 0 0) 0x7fe53c0af940 con 0x7fe53c0b4d60 2024-03-25T18:18:34.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.557+0000 7fe533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.557+0000 7fe533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.557+0000 7fe533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.557+0000 7fe533fff640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:18:34.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.557+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528003520 con 0x7fe53c0b4d60 2024-03-25T18:18:34.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328559216 0 0) 0x7fe53c0b0d50 con 0x7fe53c0b1730 2024-03-25T18:18:34.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:18:34.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.560+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c0af940 con 0x7fe53c0b1730 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071550980 0 0) 0x7fe53c0b0a70 con 0x7fe53c14be90 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:18:34.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.570+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c0b0d50 con 0x7fe53c14be90 2024-03-25T18:18:34.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1693226836 0 0) 0x7fe53c0af940 con 0x7fe53c0b1730 2024-03-25T18:18:34.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 >> v1:172.21.15.171:6789/0 conn(0x7fe53c14be90 legacy=0x7fe53c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:34.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 >> v1:172.21.15.87:6789/0 conn(0x7fe53c0b4d60 legacy=0x7fe53c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:34.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 monclient: found mon.noname-b 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0b1030 con 0x7fe53c0b1730 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe52c001670 con 0x7fe53c0b1730 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe533fff640 10 monclient: dump: 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:34.581 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe543443640 5 monclient: authenticate success, global_id 4403 2024-03-25T18:18:34.582 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.579+0000 7fe543443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe52c0017d0 con 0x7fe53c0b1730 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 1 -- 172.21.15.87:0/237526647 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe52c001be0 con 0x7fe53c0b1730 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe543443640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe543443640 4 set_mon_vals no callback set 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:34.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.600+0000 7fe533fff640 10 monclient: dump: 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:34.602 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 10 monclient: shutdown 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 1 -- 172.21.15.87:0/237526647 >> v1:172.21.15.87:6790/0 conn(0x7fe53c0b1730 legacy=0x7fe53c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 1 -- 172.21.15.87:0/237526647 shutdown_connections 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 1 -- 172.21.15.87:0/237526647 >> 172.21.15.87:0/237526647 conn(0x7fe53c0617a0 msgr2=0x7fe53c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 1 -- 172.21.15.87:0/237526647 shutdown_connections 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.601+0000 7fe543443640 1 -- 172.21.15.87:0/237526647 wait complete. 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.602+0000 7fe543443640 10 monclient: build_initial_monmap 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.602+0000 7fe543443640 10 monclient: monmap: 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:34.603 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:34.604 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.602+0000 7fe543443640 1 librados: starting msgr at 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.602+0000 7fe543443640 1 librados: starting objecter 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 1 Processor -- start 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 1 -- start start 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 1 librados: setting wanted keys 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 1 librados: calling monclient init 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 10 monclient: init 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.603+0000 7fe543443640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c14be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c1dad30 con 0x7fe53c0b4d60 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c143900 con 0x7fe53c0b1730 2024-03-25T18:18:34.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c140380 con 0x7fe53c14be90 2024-03-25T18:18:34.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:34.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.604+0000 7fe543443640 10 monclient(hunting): authenticate will time out at 8029.899247s 2024-03-25T18:18:34.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.611+0000 7fe5411df640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe53c0b1730 0x7fe53c141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52884/0 (socket says 172.21.15.87:52884) 2024-03-25T18:18:34.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.611+0000 7fe5411df640 1 -- 172.21.15.87:0/2416198888 learned_addr learned my addr 172.21.15.87:0/2416198888 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:34.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.613+0000 7fe531ffb640 10 client.?.objecter ms_handle_connect 0x7fe53c14be90 2024-03-25T18:18:34.614 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.613+0000 7fe531ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:34.616 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.615+0000 7fe531ffb640 10 client.?.objecter ms_handle_connect 0x7fe53c0b1730 2024-03-25T18:18:34.616 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.615+0000 7fe531ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:34.620 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.619+0000 7fe531ffb640 10 client.?.objecter ms_handle_connect 0x7fe53c0b4d60 2024-03-25T18:18:34.620 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.619+0000 7fe531ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010883560 0 0) 0x7fe53c140380 con 0x7fe53c14be90 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:18:34.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.623+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe520004eb0 con 0x7fe53c14be90 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802201416 0 0) 0x7fe53c143900 con 0x7fe53c0b1730 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c140380 con 0x7fe53c0b1730 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224452956 0 0) 0x7fe53c1dad30 con 0x7fe53c0b4d60 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:34.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:18:34.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.634+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c143900 con 0x7fe53c0b4d60 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057277853 0 0) 0x7fe520004eb0 con 0x7fe53c14be90 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe53c1dad30 con 0x7fe53c14be90 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe534001bd0 con 0x7fe53c14be90 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.644 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.643+0000 7fe531ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.650+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531350746 0 0) 0x7fe53c1dad30 con 0x7fe53c14be90 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 >> v1:172.21.15.171:6789/0 conn(0x7fe53c0b1730 legacy=0x7fe53c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 >> v1:172.21.15.87:6789/0 conn(0x7fe53c0b4d60 legacy=0x7fe53c132e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 1 monclient: found mon.c 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c141d50 con 0x7fe53c14be90 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe531ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 10 monclient: _renew_subs 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:18:34.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c1408a0 con 0x7fe53c14be90 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 10 monclient: _renew_subs 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c1336f0 con 0x7fe53c14be90 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.651+0000 7fe543443640 1 librados: init done 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.652+0000 7fe543443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.652+0000 7fe543443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:34.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.652+0000 7fe543443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:18:34.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.652+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c141ee0 con 0x7fe53c14be90 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe534002060 con 0x7fe53c14be90 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fe534003bf0 con 0x7fe53c14be90 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe531ffb640 10 monclient: dump: 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:34.970 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.969+0000 7fe5337fe640 4 set_mon_vals no callback set 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.970+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fe534003e10 con 0x7fe53c14be90 2024-03-25T18:18:34.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.970+0000 7fe5411df640 10 monclient: get_auth_request con 0x7fe52006ddf0 auth_method 0 2024-03-25T18:18:34.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.971+0000 7fe531ffb640 10 client.4406.objecter ms_handle_connect 0x7fe52006ddf0 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fe534053840 con 0x7fe53c14be90 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 10 client.4406.objecter ms_dispatch 0x7fe53c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe53404f350 con 0x7fe53c14be90 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:34.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:34.976+0000 7fe531ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:35.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.227+0000 7fe543443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:35.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.227+0000 7fe543443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:35.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.227+0000 7fe543443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:18:35.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.227+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe53c0627b0 con 0x7fe53c14be90 2024-03-25T18:18:35.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.250+0000 7fe531ffb640 1 -- 172.21.15.87:0/2416198888 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fe534051280 con 0x7fe53c14be90 2024-03-25T18:18:35.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.250+0000 7fe531ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:35.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.250+0000 7fe531ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:35.251 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:35.251 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:35.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 10 librados: watch_flush enter 2024-03-25T18:18:35.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 10 librados: watch_flush exit 2024-03-25T18:18:35.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 10 client.4406.objecter successfully canceled tick 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 >> v1:172.21.15.171:6800/3417775508 conn(0x7fe52006ddf0 legacy=0x7fe5200702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 10 monclient: shutdown 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 >> v1:172.21.15.87:6790/0 conn(0x7fe53c14be90 legacy=0x7fe53c1db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 shutdown_connections 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 >> 172.21.15.87:0/2416198888 conn(0x7fe53c0617a0 msgr2=0x7fe53c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 shutdown_connections 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 -- 172.21.15.87:0/2416198888 wait complete. 2024-03-25T18:18:35.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:35.252+0000 7fe543443640 1 librados: shutdown 2024-03-25T18:18:35.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:42.072 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.185+0000 7f4b46379640 10 monclient: get_monmap_and_config 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.185+0000 7f4b46379640 10 monclient: build_initial_monmap 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.185+0000 7f4b46379640 10 monclient: monmap: 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:42.187026+0000 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:42.187026+0000 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:42.187 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:42.188 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:42.188 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:42.188 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 1 Processor -- start 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 1 -- start start 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 10 monclient: init 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.186+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-a con 0x7f4b400a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-b con 0x7f4b4013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:42.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-c con 0x7f4b401430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:42.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40142520 con 0x7f4b400a98f0 2024-03-25T18:18:42.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400b28a0 con 0x7f4b401430c0 2024-03-25T18:18:42.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400b2a30 con 0x7f4b4013f710 2024-03-25T18:18:42.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:42.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.187+0000 7f4b46379640 10 monclient(hunting): authenticate will time out at 8037.482970s 2024-03-25T18:18:42.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.199+0000 7f4b3f7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4b4013f710 0x7f4b401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:56498/0 (socket says 172.21.15.87:56498) 2024-03-25T18:18:42.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.199+0000 7f4b3f7fe640 1 -- 172.21.15.87:0/4027009605 learned_addr learned my addr 172.21.15.87:0/4027009605 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.218+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878907304 0 0) 0x7f4b400b28a0 con 0x7f4b401430c0 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.219+0000 7f4b3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.219+0000 7f4b3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.219+0000 7f4b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.219+0000 7f4b3effd640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:18:42.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.219+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b28003520 con 0x7f4b401430c0 2024-03-25T18:18:42.222 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775539729 0 0) 0x7f4b40142520 con 0x7f4b400a98f0 2024-03-25T18:18:42.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:18:42.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.221+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b400b28a0 con 0x7f4b400a98f0 2024-03-25T18:18:42.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692089042 0 0) 0x7f4b400b2a30 con 0x7f4b4013f710 2024-03-25T18:18:42.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:18:42.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.223+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b40142520 con 0x7f4b4013f710 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948019600 0 0) 0x7f4b28003520 con 0x7f4b401430c0 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 >> v1:172.21.15.87:6790/0 conn(0x7f4b4013f710 legacy=0x7f4b401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 >> v1:172.21.15.87:6789/0 conn(0x7f4b400a98f0 legacy=0x7f4b400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 monclient: found mon.noname-c 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:18:42.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400b44c0 con 0x7f4b401430c0 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4b38001a10 con 0x7f4b401430c0 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b46379640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b46379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b3effd640 10 monclient: dump: 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:42.241 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:42.242 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:42.242 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:42.242 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.238+0000 7f4b46379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:42.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b38001b70 con 0x7f4b401430c0 2024-03-25T18:18:42.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:42.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 1 -- 172.21.15.87:0/4027009605 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4b38001f80 con 0x7f4b401430c0 2024-03-25T18:18:42.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b46379640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:42.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b46379640 4 set_mon_vals no callback set 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.262+0000 7f4b3effd640 10 monclient: dump: 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 10 monclient: shutdown 2024-03-25T18:18:42.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:42.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 1 -- 172.21.15.87:0/4027009605 >> v1:172.21.15.171:6789/0 conn(0x7f4b401430c0 legacy=0x7f4b4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 1 -- 172.21.15.87:0/4027009605 shutdown_connections 2024-03-25T18:18:42.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 1 -- 172.21.15.87:0/4027009605 >> 172.21.15.87:0/4027009605 conn(0x7f4b400b7ea0 msgr2=0x7f4b400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:42.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 1 -- 172.21.15.87:0/4027009605 shutdown_connections 2024-03-25T18:18:42.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.263+0000 7f4b46379640 1 -- 172.21.15.87:0/4027009605 wait complete. 2024-03-25T18:18:42.268 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.264+0000 7f4b46379640 10 monclient: build_initial_monmap 2024-03-25T18:18:42.268 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.264+0000 7f4b46379640 10 monclient: monmap: 2024-03-25T18:18:42.268 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:42.268 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.264+0000 7f4b46379640 1 librados: starting msgr at 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.264+0000 7f4b46379640 1 librados: starting objecter 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.264+0000 7f4b46379640 1 Processor -- start 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 1 -- start start 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 1 librados: setting wanted keys 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 1 librados: calling monclient init 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 10 monclient: init 2024-03-25T18:18:42.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.265+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-b con 0x7f4b400a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-c con 0x7f4b4013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 10 monclient(hunting): picked mon.noname-a con 0x7f4b401430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b401d7500 con 0x7f4b401430c0 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b401394a0 con 0x7f4b400a98f0 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40138b50 con 0x7f4b4013f710 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:42.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.266+0000 7f4b46379640 10 monclient(hunting): authenticate will time out at 8037.561976s 2024-03-25T18:18:42.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.283+0000 7f4b3ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4b400a98f0 0x7f4b40138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:42806/0 (socket says 172.21.15.87:42806) 2024-03-25T18:18:42.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.283+0000 7f4b3ffff640 1 -- 172.21.15.87:0/3347390427 learned_addr learned my addr 172.21.15.87:0/3347390427 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:42.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.284+0000 7f4b3cff9640 10 client.?.objecter ms_handle_connect 0x7f4b400a98f0 2024-03-25T18:18:42.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.284+0000 7f4b3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:42.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.287+0000 7f4b3cff9640 10 client.?.objecter ms_handle_connect 0x7f4b4013f710 2024-03-25T18:18:42.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.287+0000 7f4b3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:42.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.289+0000 7f4b3cff9640 10 client.?.objecter ms_handle_connect 0x7f4b401430c0 2024-03-25T18:18:42.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.289+0000 7f4b3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951380033 0 0) 0x7f4b401394a0 con 0x7f4b400a98f0 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:18:42.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.291+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c004eb0 con 0x7f4b400a98f0 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746056224 0 0) 0x7f4b401d7500 con 0x7f4b401430c0 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:18:42.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.293+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b401394a0 con 0x7f4b401430c0 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739886577 0 0) 0x7f4b40138b50 con 0x7f4b4013f710 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:18:42.296 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.294+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b401d7500 con 0x7f4b4013f710 2024-03-25T18:18:42.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.306+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903174640 0 0) 0x7f4b401394a0 con 0x7f4b401430c0 2024-03-25T18:18:42.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.306+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b40138b50 con 0x7f4b401430c0 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.306+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4b38001e80 con 0x7f4b401430c0 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.306+0000 7f4b3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.307+0000 7f4b3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.307+0000 7f4b3cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:42.309 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.310 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.307+0000 7f4b3cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.317+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 205869456 0 0) 0x7f4b40138b50 con 0x7f4b401430c0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.317+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 >> v1:172.21.15.87:6790/0 conn(0x7f4b4013f710 legacy=0x7f4b401d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.317+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 >> v1:172.21.15.171:6789/0 conn(0x7f4b400a98f0 legacy=0x7f4b40138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.317+0000 7f4b3cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.317+0000 7f4b3cff9640 1 monclient: found mon.a 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b401d8380 con 0x7f4b401430c0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 10 monclient: _renew_subs 2024-03-25T18:18:42.320 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40139710 con 0x7f4b401430c0 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 10 monclient: _renew_subs 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b401d7750 con 0x7f4b401430c0 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.318+0000 7f4b46379640 1 librados: init done 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.319+0000 7f4b46379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.319+0000 7f4b46379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.319+0000 7f4b46379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:42.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.319+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b400627b0 con 0x7f4b401430c0 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b38001fe0 con 0x7f4b401430c0 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4b38003dc0 con 0x7f4b401430c0 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:42.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3cff9640 10 monclient: dump: 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:42.631 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.628+0000 7f4b3e7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:42.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.629+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4b38019d90 con 0x7f4b401430c0 2024-03-25T18:18:42.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.630+0000 7f4b3ffff640 10 monclient: get_auth_request con 0x7f4b2c06ddd0 auth_method 0 2024-03-25T18:18:42.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.630+0000 7f4b3cff9640 10 client.4404.objecter ms_handle_connect 0x7f4b2c06ddd0 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4b380538e0 con 0x7f4b401430c0 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 10 client.4404.objecter ms_dispatch 0x7f4b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b3804f3f0 con 0x7f4b401430c0 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:42.640 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.638+0000 7f4b3cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:42.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.885+0000 7f4b46379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:42.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.885+0000 7f4b46379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:42.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.885+0000 7f4b46379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:18:42.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.885+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b4013fdc0 con 0x7f4b401430c0 2024-03-25T18:18:42.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.917+0000 7f4b3cff9640 1 -- 172.21.15.87:0/3347390427 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4b38051320 con 0x7f4b401430c0 2024-03-25T18:18:42.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.917+0000 7f4b3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:42.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.917+0000 7f4b3cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:42.920 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:42.920 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.920+0000 7f4b46379640 10 librados: watch_flush enter 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.920+0000 7f4b46379640 10 librados: watch_flush exit 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.920+0000 7f4b46379640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.920+0000 7f4b46379640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4b2c06ddd0 legacy=0x7f4b2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 10 monclient: shutdown 2024-03-25T18:18:42.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 >> v1:172.21.15.87:6789/0 conn(0x7f4b401430c0 legacy=0x7f4b40134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 shutdown_connections 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 >> 172.21.15.87:0/3347390427 conn(0x7f4b400b7ea0 msgr2=0x7f4b4013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 shutdown_connections 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 -- 172.21.15.87:0/3347390427 wait complete. 2024-03-25T18:18:42.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:42.921+0000 7f4b46379640 1 librados: shutdown 2024-03-25T18:18:42.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:49.739 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.850+0000 7fbc9f42e640 10 monclient: get_monmap_and_config 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.850+0000 7fbc9f42e640 10 monclient: build_initial_monmap 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.850+0000 7fbc9f42e640 10 monclient: monmap: 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:49.851610+0000 2024-03-25T18:18:49.851 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:49.851610+0000 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:49.852 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.851+0000 7fbc9f42e640 1 Processor -- start 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.851+0000 7fbc9f42e640 1 -- start start 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.851+0000 7fbc9f42e640 10 monclient: init 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc980a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc981430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:49.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc98142520 con 0x7fbc980a98f0 2024-03-25T18:18:49.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980b28a0 con 0x7fbc981430c0 2024-03-25T18:18:49.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980b2a30 con 0x7fbc9813f710 2024-03-25T18:18:49.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:49.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.852+0000 7fbc9f42e640 10 monclient(hunting): authenticate will time out at 8045.147287s 2024-03-25T18:18:49.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.863+0000 7fbc9c9c9640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fbc9813f710 0x7fbc981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:52366/0 (socket says 172.21.15.87:52366) 2024-03-25T18:18:49.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.863+0000 7fbc9c9c9640 1 -- 172.21.15.87:0/2115327664 learned_addr learned my addr 172.21.15.87:0/2115327664 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302995855 0 0) 0x7fbc980b2a30 con 0x7fbc9813f710 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:18:49.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.876+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84003520 con 0x7fbc9813f710 2024-03-25T18:18:49.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605405896 0 0) 0x7fbc98142520 con 0x7fbc980a98f0 2024-03-25T18:18:49.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:49.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:49.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:49.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:18:49.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.882+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc980b2a30 con 0x7fbc980a98f0 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826826098 0 0) 0x7fbc980b28a0 con 0x7fbc981430c0 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:18:49.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.884+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc98142520 con 0x7fbc981430c0 2024-03-25T18:18:49.896 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834762832 0 0) 0x7fbc98142520 con 0x7fbc981430c0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 >> v1:172.21.15.87:6790/0 conn(0x7fbc9813f710 legacy=0x7fbc981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 >> v1:172.21.15.87:6789/0 conn(0x7fbc980a98f0 legacy=0x7fbc980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 monclient: found mon.noname-c 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980b44c0 con 0x7fbc981430c0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fbc90001930 con 0x7fbc981430c0 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc8ffff640 10 monclient: dump: 2024-03-25T18:18:49.897 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc9f42e640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:18:49.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.895+0000 7fbc9f42e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:49.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc90001a90 con 0x7fbc981430c0 2024-03-25T18:18:49.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 1 -- 172.21.15.87:0/2115327664 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fbc90001ee0 con 0x7fbc981430c0 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc9f42e640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc9f42e640 4 set_mon_vals no callback set 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.910+0000 7fbc8ffff640 10 monclient: dump: 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:49.911 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:49.912 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 10 monclient: shutdown 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 1 -- 172.21.15.87:0/2115327664 >> v1:172.21.15.171:6789/0 conn(0x7fbc981430c0 legacy=0x7fbc9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 1 -- 172.21.15.87:0/2115327664 shutdown_connections 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 1 -- 172.21.15.87:0/2115327664 >> 172.21.15.87:0/2115327664 conn(0x7fbc980b7ea0 msgr2=0x7fbc980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 1 -- 172.21.15.87:0/2115327664 shutdown_connections 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.911+0000 7fbc9f42e640 1 -- 172.21.15.87:0/2115327664 wait complete. 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 10 monclient: build_initial_monmap 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 10 monclient: monmap: 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:49.913 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 librados: starting msgr at 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 librados: starting objecter 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 Processor -- start 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 -- start start 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 librados: setting wanted keys 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 1 librados: calling monclient init 2024-03-25T18:18:49.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.912+0000 7fbc9f42e640 10 monclient: init 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc980a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9813f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc981430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc981d7800 con 0x7fbc9813f710 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc981d83a0 con 0x7fbc980a98f0 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980b04d0 con 0x7fbc981430c0 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.913+0000 7fbc9f42e640 10 monclient(hunting): authenticate will time out at 8045.208351s 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.914+0000 7fbc9d1ca640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbc980a98f0 0x7fbc981d7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:60632/0 (socket says 172.21.15.87:60632) 2024-03-25T18:18:49.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.914+0000 7fbc9d1ca640 1 -- 172.21.15.87:0/3947012721 learned_addr learned my addr 172.21.15.87:0/3947012721 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:49.917 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.915+0000 7fbc8dffb640 10 client.?.objecter ms_handle_connect 0x7fbc980a98f0 2024-03-25T18:18:49.917 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.915+0000 7fbc8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:49.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.921+0000 7fbc8dffb640 10 client.?.objecter ms_handle_connect 0x7fbc981430c0 2024-03-25T18:18:49.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.921+0000 7fbc8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:49.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.923+0000 7fbc8dffb640 10 client.?.objecter ms_handle_connect 0x7fbc9813f710 2024-03-25T18:18:49.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.923+0000 7fbc8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152239433 0 0) 0x7fbc981d83a0 con 0x7fbc980a98f0 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:18:49.927 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.926+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7c004eb0 con 0x7fbc980a98f0 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644277972 0 0) 0x7fbc980b04d0 con 0x7fbc981430c0 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:18:49.931 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.930+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc981d83a0 con 0x7fbc981430c0 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2006902594 0 0) 0x7fbc7c004eb0 con 0x7fbc980a98f0 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc980b04d0 con 0x7fbc980a98f0 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fbc94001a10 con 0x7fbc980a98f0 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 10 monclient(hunting): dump: 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.934 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.933+0000 7fbc8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868101253 0 0) 0x7fbc981d83a0 con 0x7fbc981430c0 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc7c004eb0 con 0x7fbc981430c0 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fbc90001da0 con 0x7fbc981430c0 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.943+0000 7fbc8dffb640 10 monclient(hunting): dump: 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:49.944 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:49.945 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:49.945 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:49.945 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:49.945 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:49.945 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:49.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3638469377 0 0) 0x7fbc980b04d0 con 0x7fbc980a98f0 2024-03-25T18:18:49.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 >> v1:172.21.15.87:6790/0 conn(0x7fbc981430c0 legacy=0x7fbc98139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 >> v1:172.21.15.87:6789/0 conn(0x7fbc9813f710 legacy=0x7fbc98138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 1 monclient: found mon.b 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98139a30 con 0x7fbc980a98f0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc9f42e640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc9f42e640 10 monclient: _renew_subs 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc9f42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.945+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc980b0740 con 0x7fbc980a98f0 2024-03-25T18:18:49.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.946+0000 7fbc9f42e640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.946+0000 7fbc9f42e640 10 monclient: _renew_subs 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.946+0000 7fbc9f42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.946+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc981d7540 con 0x7fbc980a98f0 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.946+0000 7fbc9f42e640 1 librados: init done 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.947+0000 7fbc9f42e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.947+0000 7fbc9f42e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.947+0000 7fbc9f42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:49.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:49.947+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9814e1e0 con 0x7fbc980a98f0 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.219+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc94001c50 con 0x7fbc980a98f0 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fbc94003a60 con 0x7fbc980a98f0 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 10 monclient: dump: 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:50.221 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:50.222 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:50.222 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8f7fe640 4 set_mon_vals no callback set 2024-03-25T18:18:50.222 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.220+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fbc94003c80 con 0x7fbc980a98f0 2024-03-25T18:18:50.222 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.221+0000 7fbc9c9c9640 10 monclient: get_auth_request con 0x7fbc7c06de90 auth_method 0 2024-03-25T18:18:50.222 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.221+0000 7fbc8dffb640 10 client.4414.objecter ms_handle_connect 0x7fbc7c06de90 2024-03-25T18:18:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.227+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fbc94054810 con 0x7fbc980a98f0 2024-03-25T18:18:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 10 client.4414.objecter ms_dispatch 0x7fbc98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc94050320 con 0x7fbc980a98f0 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.228+0000 7fbc8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:50.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.482+0000 7fbc9f42e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:50.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.482+0000 7fbc9f42e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:50.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.482+0000 7fbc9f42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:50.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.482+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc980627b0 con 0x7fbc980a98f0 2024-03-25T18:18:50.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.507+0000 7fbc8dffb640 1 -- 172.21.15.87:0/3947012721 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fbc94052250 con 0x7fbc980a98f0 2024-03-25T18:18:50.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.507+0000 7fbc8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:50.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.507+0000 7fbc8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:50.508 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:50.508 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.509+0000 7fbc9f42e640 10 librados: watch_flush enter 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.509+0000 7fbc9f42e640 10 librados: watch_flush exit 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.509+0000 7fbc9f42e640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.509+0000 7fbc9f42e640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 >> v1:172.21.15.171:6800/3417775508 conn(0x7fbc7c06de90 legacy=0x7fbc7c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 10 monclient: shutdown 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:50.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 >> v1:172.21.15.171:6789/0 conn(0x7fbc980a98f0 legacy=0x7fbc981d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:50.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 shutdown_connections 2024-03-25T18:18:50.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 >> 172.21.15.87:0/3947012721 conn(0x7fbc980b7ea0 msgr2=0x7fbc98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:50.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 shutdown_connections 2024-03-25T18:18:50.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 -- 172.21.15.87:0/3947012721 wait complete. 2024-03-25T18:18:50.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:50.510+0000 7fbc9f42e640 1 librados: shutdown 2024-03-25T18:18:50.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:57.329 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.442+0000 7fd9e3688640 10 monclient: get_monmap_and_config 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.442+0000 7fd9e3688640 10 monclient: build_initial_monmap 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.442+0000 7fd9e3688640 10 monclient: monmap: 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:57.443 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:18:57.443526+0000 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:18:57.443526+0000 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:18:57.444 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.443+0000 7fd9e3688640 1 Processor -- start 2024-03-25T18:18:57.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.443+0000 7fd9e3688640 1 -- start start 2024-03-25T18:18:57.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.443+0000 7fd9e3688640 10 monclient: init 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-b con 0x7fd9dc0b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-a con 0x7fd9dc0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-c con 0x7fd9dc14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc0af940 con 0x7fd9dc0b1730 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc0b0a70 con 0x7fd9dc14be90 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc0b0d50 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:57.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.444+0000 7fd9e3688640 10 monclient(hunting): authenticate will time out at 8052.739522s 2024-03-25T18:18:57.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.452+0000 7fd9e1c25640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd9dc14be90 0x7fd9dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:37010/0 (socket says 172.21.15.87:37010) 2024-03-25T18:18:57.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.452+0000 7fd9e1c25640 1 -- 172.21.15.87:0/741404584 learned_addr learned my addr 172.21.15.87:0/741404584 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:57.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.459+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786265079 0 0) 0x7fd9dc0b0a70 con 0x7fd9dc14be90 2024-03-25T18:18:57.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.459+0000 7fd9d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:57.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.459+0000 7fd9d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:57.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.459+0000 7fd9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:57.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.460+0000 7fd9d3fff640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:18:57.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.460+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003520 con 0x7fd9dc14be90 2024-03-25T18:18:57.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723642506 0 0) 0x7fd9c8003520 con 0x7fd9dc14be90 2024-03-25T18:18:57.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 >> v1:172.21.15.87:6790/0 conn(0x7fd9dc0b4d60 legacy=0x7fd9dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:57.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 >> v1:172.21.15.87:6789/0 conn(0x7fd9dc0b1730 legacy=0x7fd9dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:57.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 monclient: found mon.noname-c 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc0b1030 con 0x7fd9dc14be90 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd9d4001a10 con 0x7fd9dc14be90 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9d3fff640 10 monclient: dump: 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:57.467 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9e3688640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:18:57.468 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.465+0000 7fd9e3688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9d4001be0 con 0x7fd9dc14be90 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 1 -- 172.21.15.87:0/741404584 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd9d4002030 con 0x7fd9dc14be90 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9e3688640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9e3688640 4 set_mon_vals no callback set 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:57.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.471+0000 7fd9d3fff640 10 monclient: dump: 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:57.473 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 10 monclient: shutdown 2024-03-25T18:18:57.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:57.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 1 -- 172.21.15.87:0/741404584 >> v1:172.21.15.171:6789/0 conn(0x7fd9dc14be90 legacy=0x7fd9dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 1 -- 172.21.15.87:0/741404584 shutdown_connections 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 1 -- 172.21.15.87:0/741404584 >> 172.21.15.87:0/741404584 conn(0x7fd9dc0617a0 msgr2=0x7fd9dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 1 -- 172.21.15.87:0/741404584 shutdown_connections 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.472+0000 7fd9e3688640 1 -- 172.21.15.87:0/741404584 wait complete. 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 10 monclient: build_initial_monmap 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 10 monclient: monmap: 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:18:57.476 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 librados: starting msgr at 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 librados: starting objecter 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 Processor -- start 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 -- start start 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 librados: setting wanted keys 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 1 librados: calling monclient init 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.473+0000 7fd9e3688640 10 monclient: init 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-a con 0x7fd9dc0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-b con 0x7fd9dc0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient(hunting): picked mon.noname-c con 0x7fd9dc14be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc141ce0 con 0x7fd9dc0b1730 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc141fc0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9dc140d00 con 0x7fd9dc14be90 2024-03-25T18:18:57.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:57.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.474+0000 7fd9e3688640 10 monclient(hunting): authenticate will time out at 8052.769465s 2024-03-25T18:18:57.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.480+0000 7fd9e1424640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fd9dc0b1730 0x7fd9dc1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45270/0 (socket says 172.21.15.87:45270) 2024-03-25T18:18:57.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.480+0000 7fd9e1424640 1 -- 172.21.15.87:0/1653018415 learned_addr learned my addr 172.21.15.87:0/1653018415 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:18:57.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.481+0000 7fd9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fd9dc0b1730 2024-03-25T18:18:57.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.481+0000 7fd9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:57.487 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.486+0000 7fd9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fd9dc0b4d60 2024-03-25T18:18:57.487 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.486+0000 7fd9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:57.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202546476 0 0) 0x7fd9dc141fc0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.490+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0004eb0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.491+0000 7fd9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fd9dc14be90 2024-03-25T18:18:57.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.491+0000 7fd9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233170150 0 0) 0x7fd9dc141ce0 con 0x7fd9dc0b1730 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:18:57.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.493+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9dc141fc0 con 0x7fd9dc0b1730 2024-03-25T18:18:57.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.498+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1169770102 0 0) 0x7fd9c0004eb0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9dc141ce0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd9cc001870 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:57.500 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:57.501 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.501 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.499+0000 7fd9d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:57.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678843836 0 0) 0x7fd9dc140d00 con 0x7fd9dc14be90 2024-03-25T18:18:57.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:57.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:57.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:57.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:18:57.505 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.503+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0004eb0 con 0x7fd9dc14be90 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3818634330 0 0) 0x7fd9dc141ce0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 >> v1:172.21.15.87:6790/0 conn(0x7fd9dc14be90 legacy=0x7fd9dc1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 >> v1:172.21.15.87:6789/0 conn(0x7fd9dc0b1730 legacy=0x7fd9dc1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 1 monclient: found mon.b 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc1423e0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:57.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 10 monclient: _renew_subs 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc13b470 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 10 monclient: _renew_subs 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc1d8540 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.514+0000 7fd9e3688640 1 librados: init done 2024-03-25T18:18:57.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.515+0000 7fd9e3688640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:57.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.515+0000 7fd9e3688640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:57.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.515+0000 7fd9e3688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:57.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.515+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc13b6b0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9cc0035f0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd9cc001d80 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d1ffb640 10 monclient: dump: 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:18:57.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.793+0000 7fd9d37fe640 4 set_mon_vals no callback set 2024-03-25T18:18:57.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.794+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd9cc019650 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.795+0000 7fd9e1424640 10 monclient: get_auth_request con 0x7fd9c006ddd0 auth_method 0 2024-03-25T18:18:57.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.795+0000 7fd9d1ffb640 10 client.4420.objecter ms_handle_connect 0x7fd9c006ddd0 2024-03-25T18:18:57.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.799+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd9cc053460 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.799+0000 7fd9d1ffb640 10 client.4420.objecter ms_dispatch 0x7fd9dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.799+0000 7fd9d1ffb640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.799+0000 7fd9d1ffb640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.800+0000 7fd9d1ffb640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.800+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9cc04eda0 con 0x7fd9dc0b4d60 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.800+0000 7fd9d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:57.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:57.800+0000 7fd9d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:58.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.041+0000 7fd9e3688640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:58.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.042+0000 7fd9e3688640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:58.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.042+0000 7fd9e3688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:18:58.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.042+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9dc0627b0 con 0x7fd9dc0b4d60 2024-03-25T18:18:58.153 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.152+0000 7fd9d1ffb640 1 -- 172.21.15.87:0/1653018415 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd9cc050ea0 con 0x7fd9dc0b4d60 2024-03-25T18:18:58.153 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.152+0000 7fd9d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:58.153 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.152+0000 7fd9d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:58.154 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:18:58.154 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 10 librados: watch_flush enter 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 10 librados: watch_flush exit 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd9c006ddd0 legacy=0x7fd9c0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 10 monclient: shutdown 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.154+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 >> v1:172.21.15.171:6789/0 conn(0x7fd9dc0b4d60 legacy=0x7fd9dc1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:58.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.155+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 shutdown_connections 2024-03-25T18:18:58.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.155+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 >> 172.21.15.87:0/1653018415 conn(0x7fd9dc0617a0 msgr2=0x7fd9dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:58.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.155+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 shutdown_connections 2024-03-25T18:18:58.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.155+0000 7fd9e3688640 1 -- 172.21.15.87:0/1653018415 wait complete. 2024-03-25T18:18:58.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:18:58.155+0000 7fd9e3688640 1 librados: shutdown 2024-03-25T18:18:58.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:04.973 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:05.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.090+0000 7fa679725640 10 monclient: get_monmap_and_config 2024-03-25T18:19:05.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.090+0000 7fa679725640 10 monclient: build_initial_monmap 2024-03-25T18:19:05.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.090+0000 7fa679725640 10 monclient: monmap: 2024-03-25T18:19:05.091 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:05.091505+0000 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:05.091505+0000 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:05.092 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 1 Processor -- start 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 1 -- start start 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient: init 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient(hunting): picked mon.noname-a con 0x7fa674061890 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient(hunting): picked mon.noname-b con 0x7fa67406e740 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.091+0000 7fa679725640 10 monclient(hunting): picked mon.noname-c con 0x7fa6740ae280 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.092+0000 7fa679725640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740ae830 con 0x7fa674061890 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.092+0000 7fa679725640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67406f720 con 0x7fa6740ae280 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.092+0000 7fa679725640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740ada80 con 0x7fa67406e740 2024-03-25T18:19:05.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.092+0000 7fa679725640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:05.094 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.092+0000 7fa679725640 10 monclient(hunting): authenticate will time out at 8060.386892s 2024-03-25T18:19:05.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.099+0000 7fa672ffd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa674061890 0x7fa67413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:32830/0 (socket says 172.21.15.87:32830) 2024-03-25T18:19:05.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.099+0000 7fa672ffd640 1 -- 172.21.15.87:0/3827718609 learned_addr learned my addr 172.21.15.87:0/3827718609 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245035403 0 0) 0x7fa6740ae830 con 0x7fa674061890 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:19:05.108 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.106+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa65c003520 con 0x7fa674061890 2024-03-25T18:19:05.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973012310 0 0) 0x7fa65c003520 con 0x7fa674061890 2024-03-25T18:19:05.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 >> v1:172.21.15.87:6790/0 conn(0x7fa67406e740 legacy=0x7fa674145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 >> v1:172.21.15.171:6789/0 conn(0x7fa6740ae280 legacy=0x7fa674145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 monclient: found mon.noname-a 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67413fbd0 con 0x7fa674061890 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa664001690 con 0x7fa674061890 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa679725640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa679725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa671ffb640 10 monclient: dump: 2024-03-25T18:19:05.119 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.117+0000 7fa679725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6640017f0 con 0x7fa674061890 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 1 -- 172.21.15.87:0/3827718609 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa664001c00 con 0x7fa674061890 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa679725640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa679725640 4 set_mon_vals no callback set 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.135+0000 7fa671ffb640 10 monclient: dump: 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:05.137 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 10 monclient: shutdown 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 1 -- 172.21.15.87:0/3827718609 >> v1:172.21.15.87:6789/0 conn(0x7fa674061890 legacy=0x7fa67413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 1 -- 172.21.15.87:0/3827718609 shutdown_connections 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 1 -- 172.21.15.87:0/3827718609 >> 172.21.15.87:0/3827718609 conn(0x7fa6740b0ef0 msgr2=0x7fa6740b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 1 -- 172.21.15.87:0/3827718609 shutdown_connections 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.136+0000 7fa679725640 1 -- 172.21.15.87:0/3827718609 wait complete. 2024-03-25T18:19:05.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 10 monclient: build_initial_monmap 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 10 monclient: monmap: 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 librados: starting msgr at 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 librados: starting objecter 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 Processor -- start 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 -- start start 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 librados: setting wanted keys 2024-03-25T18:19:05.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 1 librados: calling monclient init 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.137+0000 7fa679725640 10 monclient: init 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient(hunting): picked mon.noname-a con 0x7fa674061890 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient(hunting): picked mon.noname-c con 0x7fa6740ae280 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient(hunting): picked mon.noname-b con 0x7fa674142e60 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674135a60 con 0x7fa674061890 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674135cd0 con 0x7fa674142e60 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674134b00 con 0x7fa6740ae280 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:05.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.138+0000 7fa679725640 10 monclient(hunting): authenticate will time out at 8060.433584s 2024-03-25T18:19:05.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.143+0000 7fa672ffd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa674061890 0x7fa6741384d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:32882/0 (socket says 172.21.15.87:32882) 2024-03-25T18:19:05.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.143+0000 7fa672ffd640 1 -- 172.21.15.87:0/2531261454 learned_addr learned my addr 172.21.15.87:0/2531261454 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:05.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.146+0000 7fa65b7fe640 10 client.?.objecter ms_handle_connect 0x7fa674061890 2024-03-25T18:19:05.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.146+0000 7fa65b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:05.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.148+0000 7fa65b7fe640 10 client.?.objecter ms_handle_connect 0x7fa6740ae280 2024-03-25T18:19:05.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.148+0000 7fa65b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:05.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436514286 0 0) 0x7fa674135a60 con 0x7fa674061890 2024-03-25T18:19:05.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:19:05.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.157+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660004eb0 con 0x7fa674061890 2024-03-25T18:19:05.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.159+0000 7fa65b7fe640 10 client.?.objecter ms_handle_connect 0x7fa674142e60 2024-03-25T18:19:05.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.159+0000 7fa65b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:05.163 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736042893 0 0) 0x7fa674134b00 con 0x7fa6740ae280 2024-03-25T18:19:05.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:19:05.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.163+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa674135a60 con 0x7fa6740ae280 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556715850 0 0) 0x7fa660004eb0 con 0x7fa674061890 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa674134b00 con 0x7fa674061890 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa664001c00 con 0x7fa674061890 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:05.171 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.170+0000 7fa65b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143230516 0 0) 0x7fa674135cd0 con 0x7fa674142e60 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:19:05.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.171+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660004eb0 con 0x7fa674142e60 2024-03-25T18:19:05.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 535581710 0 0) 0x7fa674135a60 con 0x7fa6740ae280 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa674135cd0 con 0x7fa6740ae280 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa6680018b0 con 0x7fa6740ae280 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.183+0000 7fa65b7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:05.184 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1528256544 0 0) 0x7fa674134b00 con 0x7fa674061890 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 >> v1:172.21.15.87:6790/0 conn(0x7fa6740ae280 legacy=0x7fa6741dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 >> v1:172.21.15.171:6789/0 conn(0x7fa674142e60 legacy=0x7fa6741db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 1 monclient: found mon.a 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.194+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6741d3f70 con 0x7fa674061890 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa65b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa65b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa65b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:05.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 10 monclient: _renew_subs 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa674142a10 con 0x7fa674061890 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 10 monclient: _renew_subs 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa674138dc0 con 0x7fa674061890 2024-03-25T18:19:05.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.195+0000 7fa679725640 1 librados: init done 2024-03-25T18:19:05.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.196+0000 7fa679725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:05.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.196+0000 7fa679725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:05.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.196+0000 7fa679725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.198 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.196+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa674143250 con 0x7fa674061890 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa664004a90 con 0x7fa674061890 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa664004e20 con 0x7fa674061890 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa65b7fe640 10 monclient: dump: 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:05.515 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.513+0000 7fa6717fa640 4 set_mon_vals no callback set 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.514+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa664005040 con 0x7fa674061890 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.515+0000 7fa6727fc640 10 monclient: get_auth_request con 0x7fa66006dfd0 auth_method 0 2024-03-25T18:19:05.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.515+0000 7fa65b7fe640 10 client.4419.objecter ms_handle_connect 0x7fa66006dfd0 2024-03-25T18:19:05.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fa664054970 con 0x7fa674061890 2024-03-25T18:19:05.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 10 client.4419.objecter ms_dispatch 0x7fa674000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:05.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:05.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:05.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T18:19:05.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa664050480 con 0x7fa674061890 2024-03-25T18:19:05.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:05.522 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.520+0000 7fa65b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:05.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.773+0000 7fa679725640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:05.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.773+0000 7fa679725640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:05.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.773+0000 7fa679725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:05.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.773+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6740b0a50 con 0x7fa674061890 2024-03-25T18:19:05.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.807+0000 7fa65b7fe640 1 -- 172.21.15.87:0/2531261454 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fa6640523b0 con 0x7fa674061890 2024-03-25T18:19:05.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.807+0000 7fa65b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:05.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.807+0000 7fa65b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:05.808 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:05.808 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 10 librados: watch_flush enter 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 10 librados: watch_flush exit 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 10 client.4419.objecter successfully canceled tick 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa66006dfd0 legacy=0x7fa660070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 10 monclient: shutdown 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 >> v1:172.21.15.87:6789/0 conn(0x7fa674061890 legacy=0x7fa6741384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 shutdown_connections 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 >> 172.21.15.87:0/2531261454 conn(0x7fa6740b0ef0 msgr2=0x7fa67413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:05.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.809+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 shutdown_connections 2024-03-25T18:19:05.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.810+0000 7fa679725640 1 -- 172.21.15.87:0/2531261454 wait complete. 2024-03-25T18:19:05.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:05.810+0000 7fa679725640 1 librados: shutdown 2024-03-25T18:19:05.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:07.113 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:54.826 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.169 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:54.882 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.212 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:04:54.925 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.230 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:54.909 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.235 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:54.913 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.239 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:54.952 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.244 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:54.956 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.259 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:04:54.938 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.275 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:04:54.988 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.627 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.739+0000 7eff86eba640 10 monclient: get_monmap_and_config 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.739+0000 7eff86eba640 10 monclient: build_initial_monmap 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.739+0000 7eff86eba640 10 monclient: monmap: 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:12.740398+0000 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:12.740398+0000 2024-03-25T18:19:12.740 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:12.741 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:12.741 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:12.741 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:12.741 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:12.741 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 1 Processor -- start 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 1 -- start start 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 10 monclient: init 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-a con 0x7eff800a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.740+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-b con 0x7eff8013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-c con 0x7eff801430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:12.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff80142520 con 0x7eff800a98f0 2024-03-25T18:19:12.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800b28a0 con 0x7eff801430c0 2024-03-25T18:19:12.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800b2a30 con 0x7eff8013f710 2024-03-25T18:19:12.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:12.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.741+0000 7eff86eba640 10 monclient(hunting): authenticate will time out at 8068.035929s 2024-03-25T18:19:12.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.758+0000 7eff7ffff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7eff8013f710 0x7eff801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40696/0 (socket says 172.21.15.87:40696) 2024-03-25T18:19:12.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.758+0000 7eff7ffff640 1 -- 172.21.15.87:0/3908907156 learned_addr learned my addr 172.21.15.87:0/3908907156 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512348646 0 0) 0x7eff800b2a30 con 0x7eff8013f710 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:19:12.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.771+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70003520 con 0x7eff8013f710 2024-03-25T18:19:12.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373534358 0 0) 0x7eff80142520 con 0x7eff800a98f0 2024-03-25T18:19:12.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:12.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:12.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:12.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:19:12.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.773+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff800b2a30 con 0x7eff800a98f0 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670673433 0 0) 0x7eff800b2a30 con 0x7eff800a98f0 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 >> v1:172.21.15.87:6790/0 conn(0x7eff8013f710 legacy=0x7eff801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 >> v1:172.21.15.171:6789/0 conn(0x7eff801430c0 legacy=0x7eff8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:19:12.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff800b44c0 con 0x7eff800a98f0 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7eff74001670 con 0x7eff800a98f0 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff7f7fe640 10 monclient: dump: 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:12.786 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:12.787 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:12.787 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:12.787 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff86eba640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:19:12.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.784+0000 7eff86eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff740017d0 con 0x7eff800a98f0 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 1 -- 172.21.15.87:0/3908907156 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7eff74001c20 con 0x7eff800a98f0 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff86eba640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff86eba640 4 set_mon_vals no callback set 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:12.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.795+0000 7eff7f7fe640 10 monclient: dump: 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:12.797 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 10 monclient: shutdown 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 1 -- 172.21.15.87:0/3908907156 >> v1:172.21.15.87:6789/0 conn(0x7eff800a98f0 legacy=0x7eff800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 1 -- 172.21.15.87:0/3908907156 shutdown_connections 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 1 -- 172.21.15.87:0/3908907156 >> 172.21.15.87:0/3908907156 conn(0x7eff800b7ea0 msgr2=0x7eff800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 1 -- 172.21.15.87:0/3908907156 shutdown_connections 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.796+0000 7eff86eba640 1 -- 172.21.15.87:0/3908907156 wait complete. 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.797+0000 7eff86eba640 10 monclient: build_initial_monmap 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.797+0000 7eff86eba640 10 monclient: monmap: 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:12.799 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.797+0000 7eff86eba640 1 librados: starting msgr at 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.797+0000 7eff86eba640 1 librados: starting objecter 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.798+0000 7eff86eba640 1 Processor -- start 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.798+0000 7eff86eba640 1 -- start start 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.798+0000 7eff86eba640 1 librados: setting wanted keys 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.798+0000 7eff86eba640 1 librados: calling monclient init 2024-03-25T18:19:12.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.798+0000 7eff86eba640 10 monclient: init 2024-03-25T18:19:12.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:12.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:12.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-a con 0x7eff800a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:12.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-c con 0x7eff8013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient(hunting): picked mon.noname-b con 0x7eff801430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800af080 con 0x7eff800a98f0 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800ad240 con 0x7eff801430c0 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800b0b10 con 0x7eff8013f710 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:12.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.799+0000 7eff86eba640 10 monclient(hunting): authenticate will time out at 8068.094334s 2024-03-25T18:19:12.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.813+0000 7eff7ffff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7eff8013f710 0x7eff800abb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40736/0 (socket says 172.21.15.87:40736) 2024-03-25T18:19:12.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.813+0000 7eff7ffff640 1 -- 172.21.15.87:0/37430042 learned_addr learned my addr 172.21.15.87:0/37430042 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:12.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.816+0000 7eff7d7fa640 10 client.?.objecter ms_handle_connect 0x7eff8013f710 2024-03-25T18:19:12.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.816+0000 7eff7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:12.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.822+0000 7eff7d7fa640 10 client.?.objecter ms_handle_connect 0x7eff801430c0 2024-03-25T18:19:12.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.822+0000 7eff7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:12.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.823+0000 7eff7d7fa640 10 client.?.objecter ms_handle_connect 0x7eff800a98f0 2024-03-25T18:19:12.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.823+0000 7eff7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:12.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.825+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845961259 0 0) 0x7eff800b0b10 con 0x7eff8013f710 2024-03-25T18:19:12.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.825+0000 7eff7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:12.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.825+0000 7eff7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:12.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.825+0000 7eff7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:12.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.825+0000 7eff7d7fa640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:19:12.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.826+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff64004eb0 con 0x7eff8013f710 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239537029 0 0) 0x7eff800af080 con 0x7eff800a98f0 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:19:12.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.837+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff800b0b10 con 0x7eff800a98f0 2024-03-25T18:19:12.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605790529 0 0) 0x7eff800ad240 con 0x7eff801430c0 2024-03-25T18:19:12.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:12.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:12.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:12.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:19:12.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.844+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff800af080 con 0x7eff801430c0 2024-03-25T18:19:12.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075002151 0 0) 0x7eff64004eb0 con 0x7eff8013f710 2024-03-25T18:19:12.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff800ad240 con 0x7eff8013f710 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7eff6c001990 con 0x7eff8013f710 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 10 monclient(hunting): dump: 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:12.854 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.852+0000 7eff7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665073567 0 0) 0x7eff800b0b10 con 0x7eff800a98f0 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff64004eb0 con 0x7eff800a98f0 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7eff74001b20 con 0x7eff800a98f0 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.862+0000 7eff7d7fa640 10 monclient(hunting): dump: 2024-03-25T18:19:12.863 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:12.864 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3022373389 0 0) 0x7eff800ad240 con 0x7eff8013f710 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 >> v1:172.21.15.171:6789/0 conn(0x7eff801430c0 legacy=0x7eff801e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 >> v1:172.21.15.87:6789/0 conn(0x7eff800a98f0 legacy=0x7eff800ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 1 monclient: found mon.c 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff800af7f0 con 0x7eff8013f710 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:12.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff86eba640 5 monclient: authenticate success, global_id 4436 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff86eba640 10 monclient: _renew_subs 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff86eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.865+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff800afa40 con 0x7eff8013f710 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.866+0000 7eff86eba640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.866+0000 7eff86eba640 10 monclient: _renew_subs 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.866+0000 7eff86eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.866+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff800b1480 con 0x7eff8013f710 2024-03-25T18:19:12.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.866+0000 7eff86eba640 1 librados: init done 2024-03-25T18:19:12.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.867+0000 7eff86eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:12.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.867+0000 7eff86eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:12.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.867+0000 7eff86eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:12.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:12.867+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff800b0910 con 0x7eff8013f710 2024-03-25T18:19:13.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.092+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff6c0047e0 con 0x7eff8013f710 2024-03-25T18:19:13.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.092+0000 7eff7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:13.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.092+0000 7eff7effd640 4 set_mon_vals no callback set 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.168+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7eff6c004af0 con 0x7eff8013f710 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.168+0000 7eff7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.168+0000 7eff7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.168+0000 7eff7d7fa640 10 monclient: dump: 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:13.169 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:13.170 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:13.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.169+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7eff6c01ab40 con 0x7eff8013f710 2024-03-25T18:19:13.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.170+0000 7eff84c56640 10 monclient: get_auth_request con 0x7eff6406df70 auth_method 0 2024-03-25T18:19:13.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.170+0000 7eff7d7fa640 10 client.4436.objecter ms_handle_connect 0x7eff6406df70 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7eff6c054910 con 0x7eff8013f710 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 10 client.4436.objecter ms_dispatch 0x7eff80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff6c050250 con 0x7eff8013f710 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:13.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.175+0000 7eff7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:13.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.439+0000 7eff86eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:13.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.439+0000 7eff86eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:13.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.439+0000 7eff86eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:13.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.439+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff80074cc0 con 0x7eff8013f710 2024-03-25T18:19:13.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.466+0000 7eff7d7fa640 1 -- 172.21.15.87:0/37430042 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7eff6c052350 con 0x7eff8013f710 2024-03-25T18:19:13.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.466+0000 7eff7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:13.467 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.466+0000 7eff7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:13.467 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:13.467 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:13.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 10 librados: watch_flush enter 2024-03-25T18:19:13.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 10 librados: watch_flush exit 2024-03-25T18:19:13.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 10 client.4436.objecter successfully canceled tick 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 >> v1:172.21.15.171:6800/3417775508 conn(0x7eff6406df70 legacy=0x7eff64070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 10 monclient: shutdown 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 >> v1:172.21.15.87:6790/0 conn(0x7eff8013f710 legacy=0x7eff800abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 shutdown_connections 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 >> 172.21.15.87:0/37430042 conn(0x7eff800b7ea0 msgr2=0x7eff80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 shutdown_connections 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.468+0000 7eff86eba640 1 -- 172.21.15.87:0/37430042 wait complete. 2024-03-25T18:19:13.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:13.469+0000 7eff86eba640 1 librados: shutdown 2024-03-25T18:19:13.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:20.288 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:20.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.403+0000 7fb5298f6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:20.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.403+0000 7fb5298f6640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.403+0000 7fb5298f6640 10 monclient: monmap: 2024-03-25T18:19:20.404 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:20.404 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:20.404422+0000 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:20.404422+0000 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:20.405 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.404+0000 7fb5298f6640 1 Processor -- start 2024-03-25T18:19:20.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.404+0000 7fb5298f6640 1 -- start start 2024-03-25T18:19:20.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.404+0000 7fb5298f6640 10 monclient: init 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5240a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb52413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5241430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb524142520 con 0x7fb5241430c0 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5240b28a0 con 0x7fb52413f710 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5240b2a30 con 0x7fb5240a98f0 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.405+0000 7fb5298f6640 10 monclient(hunting): authenticate will time out at 8075.700336s 2024-03-25T18:19:20.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.418+0000 7fb522ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fb5240a98f0 0x7fb5240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:35170/0 (socket says 172.21.15.87:35170) 2024-03-25T18:19:20.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.418+0000 7fb522ffd640 1 -- 172.21.15.87:0/3887203390 learned_addr learned my addr 172.21.15.87:0/3887203390 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015140010 0 0) 0x7fb5240b2a30 con 0x7fb5240a98f0 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:19:20.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.428+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb510003520 con 0x7fb5240a98f0 2024-03-25T18:19:20.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746258045 0 0) 0x7fb5240b28a0 con 0x7fb52413f710 2024-03-25T18:19:20.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:19:20.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.432+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5240b2a30 con 0x7fb52413f710 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2171265508 0 0) 0x7fb510003520 con 0x7fb5240a98f0 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 >> v1:172.21.15.171:6789/0 conn(0x7fb52413f710 legacy=0x7fb5241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 >> v1:172.21.15.87:6789/0 conn(0x7fb5241430c0 legacy=0x7fb52414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 monclient: found mon.noname-b 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5240b44c0 con 0x7fb5240a98f0 2024-03-25T18:19:20.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb514001690 con 0x7fb5240a98f0 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5298f6640 5 monclient: authenticate success, global_id 4439 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5298f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.439+0000 7fb5227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.440+0000 7fb5227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.440+0000 7fb5227fc640 10 monclient: dump: 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.442 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.443 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.443 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.443 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.440+0000 7fb5298f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5140017f0 con 0x7fb5240a98f0 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 1 -- 172.21.15.87:0/3887203390 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb514001c00 con 0x7fb5240a98f0 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5298f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5298f6640 4 set_mon_vals no callback set 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.448+0000 7fb5227fc640 10 monclient: dump: 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.449 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.450 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.450 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.450 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.450 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.450 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 10 monclient: shutdown 2024-03-25T18:19:20.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:20.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 1 -- 172.21.15.87:0/3887203390 >> v1:172.21.15.87:6790/0 conn(0x7fb5240a98f0 legacy=0x7fb5240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 1 -- 172.21.15.87:0/3887203390 shutdown_connections 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 1 -- 172.21.15.87:0/3887203390 >> 172.21.15.87:0/3887203390 conn(0x7fb5240b7ea0 msgr2=0x7fb5240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 1 -- 172.21.15.87:0/3887203390 shutdown_connections 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.449+0000 7fb5298f6640 1 -- 172.21.15.87:0/3887203390 wait complete. 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 10 monclient: monmap: 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:20.452 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 librados: starting msgr at 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 librados: starting objecter 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 Processor -- start 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 -- start start 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 librados: setting wanted keys 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 1 librados: calling monclient init 2024-03-25T18:19:20.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.450+0000 7fb5298f6640 10 monclient: init 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.451+0000 7fb5298f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.451+0000 7fb5298f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.451+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5240a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.451+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb52413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5241430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb524138b20 con 0x7fb5241430c0 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb524137a80 con 0x7fb5240a98f0 2024-03-25T18:19:20.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb524139880 con 0x7fb52413f710 2024-03-25T18:19:20.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.452+0000 7fb5298f6640 10 monclient(hunting): authenticate will time out at 8075.747064s 2024-03-25T18:19:20.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.463+0000 7fb51bfff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fb52413f710 0x7fb524139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:35208/0 (socket says 172.21.15.87:35208) 2024-03-25T18:19:20.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.463+0000 7fb51bfff640 1 -- 172.21.15.87:0/1298144350 learned_addr learned my addr 172.21.15.87:0/1298144350 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:20.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.464+0000 7fb520ff9640 10 client.?.objecter ms_handle_connect 0x7fb52413f710 2024-03-25T18:19:20.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.464+0000 7fb520ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.465+0000 7fb520ff9640 10 client.?.objecter ms_handle_connect 0x7fb5240a98f0 2024-03-25T18:19:20.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.465+0000 7fb520ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.469+0000 7fb520ff9640 10 client.?.objecter ms_handle_connect 0x7fb5241430c0 2024-03-25T18:19:20.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.469+0000 7fb520ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.471+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372246542 0 0) 0x7fb524138b20 con 0x7fb5241430c0 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.472+0000 7fb520ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.472+0000 7fb520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.472+0000 7fb520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.472+0000 7fb520ff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:19:20.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.472+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb504004eb0 con 0x7fb5241430c0 2024-03-25T18:19:20.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038499959 0 0) 0x7fb524137a80 con 0x7fb5240a98f0 2024-03-25T18:19:20.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.474+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524138b20 con 0x7fb5240a98f0 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812852999 0 0) 0x7fb524139880 con 0x7fb52413f710 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:19:20.476 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.475+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524137a80 con 0x7fb52413f710 2024-03-25T18:19:20.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4283079080 0 0) 0x7fb504004eb0 con 0x7fb5241430c0 2024-03-25T18:19:20.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb524139880 con 0x7fb5241430c0 2024-03-25T18:19:20.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb51c001b20 con 0x7fb5241430c0 2024-03-25T18:19:20.493 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.492+0000 7fb520ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912870405 0 0) 0x7fb524138b20 con 0x7fb5240a98f0 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb504004eb0 con 0x7fb5240a98f0 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb514001b40 con 0x7fb5240a98f0 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.498+0000 7fb520ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.499 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.500 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.500 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.500 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.500 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465708005 0 0) 0x7fb524137a80 con 0x7fb52413f710 2024-03-25T18:19:20.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb524138b20 con 0x7fb52413f710 2024-03-25T18:19:20.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb50c0019b0 con 0x7fb52413f710 2024-03-25T18:19:20.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.502+0000 7fb520ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.504 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2067870687 0 0) 0x7fb504004eb0 con 0x7fb5240a98f0 2024-03-25T18:19:20.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 >> v1:172.21.15.87:6790/0 conn(0x7fb52413f710 legacy=0x7fb524139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 >> v1:172.21.15.87:6789/0 conn(0x7fb5241430c0 legacy=0x7fb5241e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 1 monclient: found mon.b 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb524139a10 con 0x7fb5240a98f0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb520ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 10 monclient: _renew_subs 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5241dab00 con 0x7fb5240a98f0 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 10 monclient: _renew_subs 2024-03-25T18:19:20.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52413b4f0 con 0x7fb5240a98f0 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.509+0000 7fb5298f6640 1 librados: init done 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.510+0000 7fb5298f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.510+0000 7fb5298f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.510+0000 7fb5298f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:20.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.510+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb524074cc0 con 0x7fb5240a98f0 2024-03-25T18:19:20.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb514004a90 con 0x7fb5240a98f0 2024-03-25T18:19:20.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb514004da0 con 0x7fb5240a98f0 2024-03-25T18:19:20.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb520ff9640 10 monclient: dump: 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:20.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.798+0000 7fb521ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:20.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.799+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fb51401ad70 con 0x7fb5240a98f0 2024-03-25T18:19:20.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.800+0000 7fb51bfff640 10 monclient: get_auth_request con 0x7fb50406e160 auth_method 0 2024-03-25T18:19:20.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.800+0000 7fb520ff9640 10 client.4435.objecter ms_handle_connect 0x7fb50406e160 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fb514054880 con 0x7fb5240a98f0 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 10 client.4435.objecter ms_dispatch 0x7fb524000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb514050390 con 0x7fb5240a98f0 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:20.804+0000 7fb520ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:21.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.052+0000 7fb5298f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:21.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.052+0000 7fb5298f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:21.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.052+0000 7fb5298f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:21.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.052+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5241dae00 con 0x7fb5240a98f0 2024-03-25T18:19:21.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.092+0000 7fb520ff9640 1 -- 172.21.15.87:0/1298144350 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fb5140522c0 con 0x7fb5240a98f0 2024-03-25T18:19:21.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.092+0000 7fb520ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:21.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.092+0000 7fb520ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:21.093 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:21.093 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 10 librados: watch_flush enter 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 10 librados: watch_flush exit 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 >> v1:172.21.15.171:6800/3417775508 conn(0x7fb50406e160 legacy=0x7fb504070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 10 monclient: shutdown 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 >> v1:172.21.15.171:6789/0 conn(0x7fb5240a98f0 legacy=0x7fb524137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 shutdown_connections 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 >> 172.21.15.87:0/1298144350 conn(0x7fb5240b7ea0 msgr2=0x7fb524074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 shutdown_connections 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 -- 172.21.15.87:0/1298144350 wait complete. 2024-03-25T18:19:21.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:21.094+0000 7fb5298f6640 1 librados: shutdown 2024-03-25T18:19:21.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:27.194 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:14.907 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.199 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:14.913 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.228 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:14.941 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.257 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:14.969 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.264 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:14.943 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.272 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:14.950 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.277 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:14.990 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.298 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:14.976 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.305 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:15.018 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.915 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:28.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.030+0000 7fd3334a8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:28.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.030+0000 7fd3334a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:28.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.030+0000 7fd3334a8640 10 monclient: monmap: 2024-03-25T18:19:28.032 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:28.032346+0000 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:28.032346+0000 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:28.033 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.031+0000 7fd3334a8640 1 Processor -- start 2024-03-25T18:19:28.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.031+0000 7fd3334a8640 1 -- start start 2024-03-25T18:19:28.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.031+0000 7fd3334a8640 10 monclient: init 2024-03-25T18:19:28.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c0a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c13f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c1430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c142520 con 0x7fd32c0a98f0 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0b28a0 con 0x7fd32c1430c0 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0b2a30 con 0x7fd32c13f710 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:28.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.032+0000 7fd3334a8640 10 monclient(hunting): authenticate will time out at 8083.328456s 2024-03-25T18:19:28.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.037+0000 7fd331a45640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd32c1430c0 0x7fd32c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:37844/0 (socket says 172.21.15.87:37844) 2024-03-25T18:19:28.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.037+0000 7fd331a45640 1 -- 172.21.15.87:0/1522968921 learned_addr learned my addr 172.21.15.87:0/1522968921 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:28.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225002434 0 0) 0x7fd32c0b28a0 con 0x7fd32c1430c0 2024-03-25T18:19:28.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.057+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd318003520 con 0x7fd32c1430c0 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995230347 0 0) 0x7fd32c0b2a30 con 0x7fd32c13f710 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:19:28.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.058+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c0b28a0 con 0x7fd32c13f710 2024-03-25T18:19:28.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868881340 0 0) 0x7fd32c142520 con 0x7fd32c0a98f0 2024-03-25T18:19:28.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:19:28.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.060+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c0b2a30 con 0x7fd32c0a98f0 2024-03-25T18:19:28.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393783683 0 0) 0x7fd32c0b28a0 con 0x7fd32c13f710 2024-03-25T18:19:28.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 >> v1:172.21.15.171:6789/0 conn(0x7fd32c1430c0 legacy=0x7fd32c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 >> v1:172.21.15.87:6789/0 conn(0x7fd32c0a98f0 legacy=0x7fd32c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 monclient: found mon.noname-b 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0b44c0 con 0x7fd32c13f710 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd31c001670 con 0x7fd32c13f710 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd3334a8640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd3334a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd323fff640 10 monclient: dump: 2024-03-25T18:19:28.071 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.068+0000 7fd3334a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd31c0017d0 con 0x7fd32c13f710 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 1 -- 172.21.15.87:0/1522968921 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd31c001be0 con 0x7fd32c13f710 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd3334a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd3334a8640 4 set_mon_vals no callback set 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.098+0000 7fd323fff640 10 monclient: dump: 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.100 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:28.101 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 10 monclient: shutdown 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 1 -- 172.21.15.87:0/1522968921 >> v1:172.21.15.87:6790/0 conn(0x7fd32c13f710 legacy=0x7fd32c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 1 -- 172.21.15.87:0/1522968921 shutdown_connections 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 1 -- 172.21.15.87:0/1522968921 >> 172.21.15.87:0/1522968921 conn(0x7fd32c0b7ea0 msgr2=0x7fd32c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 1 -- 172.21.15.87:0/1522968921 shutdown_connections 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.099+0000 7fd3334a8640 1 -- 172.21.15.87:0/1522968921 wait complete. 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.100+0000 7fd3334a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.100+0000 7fd3334a8640 10 monclient: monmap: 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:28.102 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:28.103 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.100+0000 7fd3334a8640 1 librados: starting msgr at 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.100+0000 7fd3334a8640 1 librados: starting objecter 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.100+0000 7fd3334a8640 1 Processor -- start 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 -- start start 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 librados: setting wanted keys 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 librados: calling monclient init 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient: init 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c0a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c13f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:28.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c1430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:28.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c136fa0 con 0x7fd32c1430c0 2024-03-25T18:19:28.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c13b780 con 0x7fd32c0a98f0 2024-03-25T18:19:28.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1d7ba0 con 0x7fd32c13f710 2024-03-25T18:19:28.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:28.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.101+0000 7fd3334a8640 10 monclient(hunting): authenticate will time out at 8083.397707s 2024-03-25T18:19:28.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.108+0000 7fd331a45640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fd32c1430c0 0x7fd32c1e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:39302/0 (socket says 172.21.15.87:39302) 2024-03-25T18:19:28.110 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.108+0000 7fd331a45640 1 -- 172.21.15.87:0/1528139003 learned_addr learned my addr 172.21.15.87:0/1528139003 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:28.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.109+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c1430c0 2024-03-25T18:19:28.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.109+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:28.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.114+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c13f710 2024-03-25T18:19:28.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.114+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333965504 0 0) 0x7fd32c136fa0 con 0x7fd32c1430c0 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:19:28.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.118+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c1430c0 2024-03-25T18:19:28.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029300741 0 0) 0x7fd32c1d7ba0 con 0x7fd32c13f710 2024-03-25T18:19:28.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:19:28.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.125+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c136fa0 con 0x7fd32c13f710 2024-03-25T18:19:28.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.130+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c0a98f0 2024-03-25T18:19:28.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.130+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 569821108 0 0) 0x7fd310004eb0 con 0x7fd32c1430c0 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c1d7ba0 con 0x7fd32c1430c0 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd324001b30 con 0x7fd32c1430c0 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.139 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.137+0000 7fd321ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:28.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578614336 0 0) 0x7fd32c13b780 con 0x7fd32c0a98f0 2024-03-25T18:19:28.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:28.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:28.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:28.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:19:28.143 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.140+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c0a98f0 2024-03-25T18:19:28.151 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321880741 0 0) 0x7fd32c136fa0 con 0x7fd32c13f710 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c13b780 con 0x7fd32c13f710 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd31c001b20 con 0x7fd32c13f710 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.150+0000 7fd321ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:28.152 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1244286860 0 0) 0x7fd32c1d7ba0 con 0x7fd32c1430c0 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 >> v1:172.21.15.87:6790/0 conn(0x7fd32c13f710 legacy=0x7fd32c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 >> v1:172.21.15.171:6789/0 conn(0x7fd32c0a98f0 legacy=0x7fd32c136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 1 monclient: found mon.a 2024-03-25T18:19:28.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c137990 con 0x7fd32c1430c0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 10 monclient: _renew_subs 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c137c40 con 0x7fd32c1430c0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 10 monclient: _renew_subs 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c139860 con 0x7fd32c1430c0 2024-03-25T18:19:28.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.152+0000 7fd3334a8640 1 librados: init done 2024-03-25T18:19:28.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.153+0000 7fd3334a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:28.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.153+0000 7fd3334a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:28.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.153+0000 7fd3334a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:28.156 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.153+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c1375f0 con 0x7fd32c1430c0 2024-03-25T18:19:28.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd324001fe0 con 0x7fd32c1430c0 2024-03-25T18:19:28.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:28.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd324004d00 con 0x7fd32c1430c0 2024-03-25T18:19:28.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:28.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd321ffb640 10 monclient: dump: 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:28.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.443+0000 7fd3237fe640 4 set_mon_vals no callback set 2024-03-25T18:19:28.447 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.444+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd32401acd0 con 0x7fd32c1430c0 2024-03-25T18:19:28.447 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.445+0000 7fd331244640 10 monclient: get_auth_request con 0x7fd31006df20 auth_method 0 2024-03-25T18:19:28.447 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.445+0000 7fd321ffb640 10 client.4437.objecter ms_handle_connect 0x7fd31006df20 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd324054af0 con 0x7fd32c1430c0 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 10 client.4437.objecter ms_dispatch 0x7fd32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd324050430 con 0x7fd32c1430c0 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:28.451 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.449+0000 7fd321ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:28.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.668+0000 7fd3334a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:28.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.668+0000 7fd3334a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:28.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.668+0000 7fd3334a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:28.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.668+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd32c0627b0 con 0x7fd32c1430c0 2024-03-25T18:19:28.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.757+0000 7fd321ffb640 1 -- 172.21.15.87:0/1528139003 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd324052530 con 0x7fd32c1430c0 2024-03-25T18:19:28.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.757+0000 7fd321ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:28.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.757+0000 7fd321ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:28.759 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:28.760 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:28.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 10 librados: watch_flush enter 2024-03-25T18:19:28.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 10 librados: watch_flush exit 2024-03-25T18:19:28.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T18:19:28.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 10 client.4437.objecter successfully canceled tick 2024-03-25T18:19:28.762 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd31006df20 legacy=0x7fd3100703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 10 monclient: shutdown 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 >> v1:172.21.15.87:6789/0 conn(0x7fd32c1430c0 legacy=0x7fd32c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 shutdown_connections 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 >> 172.21.15.87:0/1528139003 conn(0x7fd32c0b7ea0 msgr2=0x7fd32c14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 shutdown_connections 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 -- 172.21.15.87:0/1528139003 wait complete. 2024-03-25T18:19:28.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:28.760+0000 7fd3334a8640 1 librados: shutdown 2024-03-25T18:19:28.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:35.580 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:35.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.693+0000 7f43ac7d7640 10 monclient: get_monmap_and_config 2024-03-25T18:19:35.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.693+0000 7f43ac7d7640 10 monclient: build_initial_monmap 2024-03-25T18:19:35.694 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.693+0000 7f43ac7d7640 10 monclient: monmap: 2024-03-25T18:19:35.694 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:35.694 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:35.694625+0000 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:35.694625+0000 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:35.695 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.694+0000 7f43ac7d7640 1 Processor -- start 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.694+0000 7f43ac7d7640 1 -- start start 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.694+0000 7f43ac7d7640 10 monclient: init 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-c con 0x7f43a4061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f43a413d9f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-a con 0x7f43a4143d50 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4071af0 con 0x7f43a4143d50 2024-03-25T18:19:35.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a41441c0 con 0x7f43a4061830 2024-03-25T18:19:35.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4072810 con 0x7f43a413d9f0 2024-03-25T18:19:35.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:35.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.695+0000 7f43ac7d7640 10 monclient(hunting): authenticate will time out at 8090.990386s 2024-03-25T18:19:35.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.697+0000 7f43aad74640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f43a4143d50 0x7f43a414b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:44068/0 (socket says 172.21.15.87:44068) 2024-03-25T18:19:35.698 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.697+0000 7f43aad74640 1 -- 172.21.15.87:0/884960452 learned_addr learned my addr 172.21.15.87:0/884960452 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051171214 0 0) 0x7f43a4071af0 con 0x7f43a4143d50 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 10 monclient(hunting): _negotiate 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:19:35.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.704+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4390003520 con 0x7f43a4143d50 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020392310 0 0) 0x7f4390003520 con 0x7f43a4143d50 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 >> v1:172.21.15.87:6790/0 conn(0x7f43a413d9f0 legacy=0x7f43a4143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 >> v1:172.21.15.171:6789/0 conn(0x7f43a4061830 legacy=0x7f43a413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: _finish_hunting 0 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 monclient: found mon.noname-a 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4072af0 con 0x7f43a4143d50 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: _finish_auth 0 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: _check_auth_tickets 2024-03-25T18:19:35.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f439c001a10 con 0x7f43a4143d50 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43a9571640 10 monclient: dump: 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43ac7d7640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:19:35.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.712+0000 7f43ac7d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f439c001be0 con 0x7f43a4143d50 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 1 -- 172.21.15.87:0/884960452 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f439c002030 con 0x7f43a4143d50 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43ac7d7640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43ac7d7640 4 set_mon_vals no callback set 2024-03-25T18:19:35.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.737+0000 7f43a9571640 10 monclient: dump: 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:35.739 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 10 monclient: shutdown 2024-03-25T18:19:35.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:35.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 1 -- 172.21.15.87:0/884960452 >> v1:172.21.15.87:6789/0 conn(0x7f43a4143d50 legacy=0x7f43a414b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 1 -- 172.21.15.87:0/884960452 shutdown_connections 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 1 -- 172.21.15.87:0/884960452 >> 172.21.15.87:0/884960452 conn(0x7f43a40b11e0 msgr2=0x7f43a40b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 1 -- 172.21.15.87:0/884960452 shutdown_connections 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.738+0000 7f43ac7d7640 1 -- 172.21.15.87:0/884960452 wait complete. 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 10 monclient: build_initial_monmap 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 10 monclient: monmap: 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:35.741 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 librados: starting msgr at 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 librados: starting objecter 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 Processor -- start 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 -- start start 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 librados: setting wanted keys 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 1 librados: calling monclient init 2024-03-25T18:19:35.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.739+0000 7f43ac7d7640 10 monclient: init 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f43a4061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-c con 0x7f43a413d9f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient(hunting): picked mon.noname-a con 0x7f43a4143d50 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a40a9ea0 con 0x7f43a4143d50 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a40aa110 con 0x7f43a4061830 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4137f70 con 0x7f43a413d9f0 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:35.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.740+0000 7f43ac7d7640 10 monclient(hunting): authenticate will time out at 8091.035436s 2024-03-25T18:19:35.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.749+0000 7f43aa573640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f43a4061830 0x7f43a413e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:42780/0 (socket says 172.21.15.87:42780) 2024-03-25T18:19:35.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.749+0000 7f43aa573640 1 -- 172.21.15.87:0/4095829023 learned_addr learned my addr 172.21.15.87:0/4095829023 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:35.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.749+0000 7f439affd640 10 client.?.objecter ms_handle_connect 0x7f43a4061830 2024-03-25T18:19:35.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.749+0000 7f439affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:35.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.751+0000 7f439affd640 10 client.?.objecter ms_handle_connect 0x7f43a413d9f0 2024-03-25T18:19:35.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.751+0000 7f439affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827211827 0 0) 0x7f43a40aa110 con 0x7f43a4061830 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:19:35.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.756+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4388004eb0 con 0x7f43a4061830 2024-03-25T18:19:35.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.758+0000 7f439affd640 10 client.?.objecter ms_handle_connect 0x7f43a4143d50 2024-03-25T18:19:35.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.758+0000 7f439affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:35.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524407768 0 0) 0x7f43a4137f70 con 0x7f43a413d9f0 2024-03-25T18:19:35.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:35.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:35.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:35.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:19:35.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.759+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a40aa110 con 0x7f43a413d9f0 2024-03-25T18:19:35.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189775351 0 0) 0x7f43a40a9ea0 con 0x7f43a4143d50 2024-03-25T18:19:35.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:35.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:35.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:35.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:19:35.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.767+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a4137f70 con 0x7f43a4143d50 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112711720 0 0) 0x7f4388004eb0 con 0x7f43a4061830 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43a40a9ea0 con 0x7f43a4061830 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f43a0001a30 con 0x7f43a4061830 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 10 monclient(hunting): dump: 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.775 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.774+0000 7f439affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5710284 0 0) 0x7f43a40aa110 con 0x7f43a413d9f0 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4388004eb0 con 0x7f43a413d9f0 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4394001510 con 0x7f43a413d9f0 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:35.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.776+0000 7f439affd640 10 monclient(hunting): dump: 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:35.778 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1367700005 0 0) 0x7f4388004eb0 con 0x7f43a413d9f0 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 >> v1:172.21.15.171:6789/0 conn(0x7f43a4061830 legacy=0x7f43a413e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 >> v1:172.21.15.87:6789/0 conn(0x7f43a4143d50 legacy=0x7f43a4137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 1 monclient: found mon.c 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:35.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a40aa3f0 con 0x7f43a413d9f0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f439affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 10 monclient: _renew_subs 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40aa6d0 con 0x7f43a413d9f0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 10 monclient: _renew_subs 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a4142c80 con 0x7f43a413d9f0 2024-03-25T18:19:35.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.783+0000 7f43ac7d7640 1 librados: init done 2024-03-25T18:19:35.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.784+0000 7f43ac7d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:35.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.784+0000 7f43ac7d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:35.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.784+0000 7f43ac7d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:35.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:35.784+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a40627b0 con 0x7f43a413d9f0 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4394001a00 con 0x7f43a413d9f0 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4394003910 con 0x7f43a413d9f0 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 10 monclient: dump: 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:36.097 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f43a8d70640 4 set_mon_vals no callback set 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.096+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f43940198e0 con 0x7f43a413d9f0 2024-03-25T18:19:36.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.097+0000 7f43aa573640 10 monclient: get_auth_request con 0x7f438806df70 auth_method 0 2024-03-25T18:19:36.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.097+0000 7f439affd640 10 client.4451.objecter ms_handle_connect 0x7f438806df70 2024-03-25T18:19:36.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4394053540 con 0x7f43a413d9f0 2024-03-25T18:19:36.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 10 client.4451.objecter ms_dispatch 0x7f43a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:36.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:36.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:36.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:19:36.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f439404f050 con 0x7f43a413d9f0 2024-03-25T18:19:36.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:36.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.103+0000 7f439affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:36.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.322+0000 7f43ac7d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:36.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.322+0000 7f43ac7d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:36.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.322+0000 7f43ac7d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:36.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.322+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43a4142f80 con 0x7f43a413d9f0 2024-03-25T18:19:36.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.358+0000 7f439affd640 1 -- 172.21.15.87:0/4095829023 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4394056ba0 con 0x7f43a413d9f0 2024-03-25T18:19:36.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.358+0000 7f439affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:36.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.358+0000 7f439affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:36.438 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:36.438 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 10 librados: watch_flush enter 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 10 librados: watch_flush exit 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 >> v1:172.21.15.171:6800/3417775508 conn(0x7f438806df70 legacy=0x7f4388070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 10 monclient: shutdown 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.360+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 >> v1:172.21.15.87:6790/0 conn(0x7f43a413d9f0 legacy=0x7f43a4137150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:36.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.361+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 shutdown_connections 2024-03-25T18:19:36.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.361+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 >> 172.21.15.87:0/4095829023 conn(0x7f43a40b11e0 msgr2=0x7f43a40af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:36.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.361+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 shutdown_connections 2024-03-25T18:19:36.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.361+0000 7f43ac7d7640 1 -- 172.21.15.87:0/4095829023 wait complete. 2024-03-25T18:19:36.441 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:36.361+0000 7f43ac7d7640 1 librados: shutdown 2024-03-25T18:19:36.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:43.248 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.363+0000 7fd35149a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.363+0000 7fd35149a640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.363+0000 7fd35149a640 10 monclient: monmap: 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:43.363936+0000 2024-03-25T18:19:43.364 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:43.363936+0000 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:43.365 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 1 Processor -- start 2024-03-25T18:19:43.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 1 -- start start 2024-03-25T18:19:43.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient: init 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-a con 0x7fd34c13fc60 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c070570 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-b con 0x7fd34c0727a0 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c13c9f0 con 0x7fd34c13fc60 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c06fac0 con 0x7fd34c070570 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c0715d0 con 0x7fd34c0727a0 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.364+0000 7fd35149a640 10 monclient(hunting): authenticate will time out at 8098.659786s 2024-03-25T18:19:43.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.376+0000 7fd34b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd34c0727a0 0x7fd34c14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:54994/0 (socket says 172.21.15.87:54994) 2024-03-25T18:19:43.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.376+0000 7fd34b7fe640 1 -- 172.21.15.87:0/2561319733 learned_addr learned my addr 172.21.15.87:0/2561319733 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224907281 0 0) 0x7fd34c13c9f0 con 0x7fd34c13fc60 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:19:43.392 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.391+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338003520 con 0x7fd34c13fc60 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658150785 0 0) 0x7fd34c0715d0 con 0x7fd34c0727a0 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:19:43.397 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.396+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c13c9f0 con 0x7fd34c0727a0 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.414+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481611109 0 0) 0x7fd34c06fac0 con 0x7fd34c070570 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.414+0000 7fd34a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.414+0000 7fd34a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.414+0000 7fd34a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.415+0000 7fd34a7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:19:43.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.415+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c0715d0 con 0x7fd34c070570 2024-03-25T18:19:43.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589674025 0 0) 0x7fd34c13c9f0 con 0x7fd34c0727a0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 >> v1:172.21.15.171:6789/0 conn(0x7fd34c070570 legacy=0x7fd34c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 >> v1:172.21.15.87:6789/0 conn(0x7fd34c13fc60 legacy=0x7fd34c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0718b0 con 0x7fd34c0727a0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd3440019f0 con 0x7fd34c0727a0 2024-03-25T18:19:43.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd34a7fc640 10 monclient: dump: 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:43.429 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd35149a640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:19:43.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.426+0000 7fd35149a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:43.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd344001b50 con 0x7fd34c0727a0 2024-03-25T18:19:43.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 1 -- 172.21.15.87:0/2561319733 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd344001f60 con 0x7fd34c0727a0 2024-03-25T18:19:43.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.456 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd34a7fc640 10 monclient: dump: 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd35149a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:43.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.455+0000 7fd35149a640 4 set_mon_vals no callback set 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 10 monclient: shutdown 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 1 -- 172.21.15.87:0/2561319733 >> v1:172.21.15.87:6790/0 conn(0x7fd34c0727a0 legacy=0x7fd34c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 1 -- 172.21.15.87:0/2561319733 shutdown_connections 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 1 -- 172.21.15.87:0/2561319733 >> 172.21.15.87:0/2561319733 conn(0x7fd34c0b4a80 msgr2=0x7fd34c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 1 -- 172.21.15.87:0/2561319733 shutdown_connections 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.456+0000 7fd35149a640 1 -- 172.21.15.87:0/2561319733 wait complete. 2024-03-25T18:19:43.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.457+0000 7fd35149a640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.457+0000 7fd35149a640 10 monclient: monmap: 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.457+0000 7fd35149a640 1 librados: starting msgr at 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.457+0000 7fd35149a640 1 librados: starting objecter 2024-03-25T18:19:43.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 Processor -- start 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 -- start start 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 librados: setting wanted keys 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 librados: calling monclient init 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient: init 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-b con 0x7fd34c070570 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-a con 0x7fd34c0727a0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient(hunting): picked mon.noname-c con 0x7fd34c13fc60 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c13c2b0 con 0x7fd34c0727a0 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c13ca20 con 0x7fd34c070570 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c0ae3c0 con 0x7fd34c13fc60 2024-03-25T18:19:43.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.458+0000 7fd35149a640 10 monclient(hunting): authenticate will time out at 8098.753538s 2024-03-25T18:19:43.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.462+0000 7fd34b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd34c13fc60 0x7fd34c1e0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55050/0 (socket says 172.21.15.87:55050) 2024-03-25T18:19:43.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.462+0000 7fd34b7fe640 1 -- 172.21.15.87:0/3359372126 learned_addr learned my addr 172.21.15.87:0/3359372126 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:43.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.465+0000 7fd3437fe640 10 client.?.objecter ms_handle_connect 0x7fd34c13fc60 2024-03-25T18:19:43.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.465+0000 7fd3437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254511255 0 0) 0x7fd34c0ae3c0 con 0x7fd34c13fc60 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:19:43.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.472+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c004eb0 con 0x7fd34c13fc60 2024-03-25T18:19:43.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.476+0000 7fd3437fe640 10 client.?.objecter ms_handle_connect 0x7fd34c070570 2024-03-25T18:19:43.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.476+0000 7fd3437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.482+0000 7fd3437fe640 10 client.?.objecter ms_handle_connect 0x7fd34c0727a0 2024-03-25T18:19:43.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.482+0000 7fd3437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839285787 0 0) 0x7fd34c13ca20 con 0x7fd34c070570 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:19:43.488 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.487+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c0ae3c0 con 0x7fd34c070570 2024-03-25T18:19:43.496 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983666030 0 0) 0x7fd34c13c2b0 con 0x7fd34c0727a0 2024-03-25T18:19:43.496 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.496 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.496 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.496 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c13ca20 con 0x7fd34c0727a0 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 657699355 0 0) 0x7fd32c004eb0 con 0x7fd34c13fc60 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd34c13c2b0 con 0x7fd34c13fc60 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd344001f20 con 0x7fd34c13fc60 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): dump: 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:43.497 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:43.498 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:43.498 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.495+0000 7fd3437fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:43.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 163241820 0 0) 0x7fd34c0ae3c0 con 0x7fd34c070570 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c004eb0 con 0x7fd34c070570 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd33c001950 con 0x7fd34c070570 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.508+0000 7fd3437fe640 10 monclient(hunting): dump: 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:43.510 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:43.511 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:43.511 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 945324899 0 0) 0x7fd34c13c2b0 con 0x7fd34c13fc60 2024-03-25T18:19:43.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 >> v1:172.21.15.171:6789/0 conn(0x7fd34c070570 legacy=0x7fd34c1d5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 >> v1:172.21.15.87:6789/0 conn(0x7fd34c0727a0 legacy=0x7fd34c134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 1 monclient: found mon.c 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0ae550 con 0x7fd34c13fc60 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd3437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 10 monclient: _renew_subs 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:43.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c1d5b50 con 0x7fd34c13fc60 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 10 monclient: _renew_subs 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c1d5ea0 con 0x7fd34c13fc60 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.511+0000 7fd35149a640 1 librados: init done 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.512+0000 7fd35149a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.512+0000 7fd35149a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.512+0000 7fd35149a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:43.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.512+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c13c000 con 0x7fd34c13fc60 2024-03-25T18:19:43.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd344004e10 con 0x7fd34c13fc60 2024-03-25T18:19:43.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd344005120 con 0x7fd34c13fc60 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd3437fe640 10 monclient: dump: 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:43.811 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.809+0000 7fd349ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.810+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd34401b0f0 con 0x7fd34c13fc60 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.811+0000 7fd34affd640 10 monclient: get_auth_request con 0x7fd32c06df70 auth_method 0 2024-03-25T18:19:43.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.811+0000 7fd3437fe640 10 client.4457.objecter ms_handle_connect 0x7fd32c06df70 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fd344054c80 con 0x7fd34c13fc60 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 10 client.4457.objecter ms_dispatch 0x7fd34c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd344050790 con 0x7fd34c13fc60 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:43.816+0000 7fd3437fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:44.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.052+0000 7fd35149a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.052+0000 7fd35149a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.052+0000 7fd35149a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:19:44.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.052+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd34c0ae6e0 con 0x7fd34c13fc60 2024-03-25T18:19:44.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.096+0000 7fd3437fe640 1 -- 172.21.15.87:0/3359372126 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fd34c0ae6e0 con 0x7fd34c13fc60 2024-03-25T18:19:44.097 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.096+0000 7fd3437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:44.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.096+0000 7fd3437fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:44.098 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:44.098 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:44.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.098+0000 7fd35149a640 10 librados: watch_flush enter 2024-03-25T18:19:44.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 10 librados: watch_flush exit 2024-03-25T18:19:44.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 10 client.4457.objecter successfully canceled tick 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd32c06df70 legacy=0x7fd32c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 10 monclient: shutdown 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 >> v1:172.21.15.87:6790/0 conn(0x7fd34c13fc60 legacy=0x7fd34c1e0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 shutdown_connections 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 >> 172.21.15.87:0/3359372126 conn(0x7fd34c0b4a80 msgr2=0x7fd34c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 shutdown_connections 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 -- 172.21.15.87:0/3359372126 wait complete. 2024-03-25T18:19:44.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:44.099+0000 7fd35149a640 1 librados: shutdown 2024-03-25T18:19:44.114 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:47.225 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:34.938 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.271 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:34.985 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.307 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:35.020 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.333 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:35.046 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.342 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:35.055 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.354 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:35.032 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.359 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:35.038 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.376 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:35.089 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:47.386 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:35.065 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:50.921 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:51.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.032+0000 7f2fbc5ab640 10 monclient: get_monmap_and_config 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.032+0000 7f2fbc5ab640 10 monclient: build_initial_monmap 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.032+0000 7f2fbc5ab640 10 monclient: monmap: 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:51.033585+0000 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:51.033585+0000 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.034 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:51.035 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:51.035 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:51.035 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.033+0000 7f2fbc5ab640 1 Processor -- start 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.033+0000 7f2fbc5ab640 1 -- start start 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.033+0000 7f2fbc5ab640 10 monclient: init 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb4061830 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb406ff10 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb413c8c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4071cd0 con 0x7f2fb4061830 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4070790 con 0x7f2fb413c8c0 2024-03-25T18:19:51.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4070e00 con 0x7f2fb406ff10 2024-03-25T18:19:51.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:51.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.034+0000 7f2fbc5ab640 10 monclient(hunting): authenticate will time out at 8106.329417s 2024-03-25T18:19:51.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.041+0000 7f2fb9b46640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f2fb406ff10 0x7f2fb40723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:34928/0 (socket says 172.21.15.87:34928) 2024-03-25T18:19:51.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.042+0000 7f2fb9b46640 1 -- 172.21.15.87:0/2965370836 learned_addr learned my addr 172.21.15.87:0/2965370836 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:51.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838372727 0 0) 0x7f2fb4070e00 con 0x7f2fb406ff10 2024-03-25T18:19:51.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:19:51.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.052+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa0003520 con 0x7f2fb406ff10 2024-03-25T18:19:51.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587422529 0 0) 0x7f2fb4071cd0 con 0x7f2fb4061830 2024-03-25T18:19:51.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4070e00 con 0x7f2fb4061830 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195526542 0 0) 0x7f2fb4070790 con 0x7f2fb413c8c0 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:19:51.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.063+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4071cd0 con 0x7f2fb413c8c0 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833473514 0 0) 0x7f2fa0003520 con 0x7f2fb406ff10 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 >> v1:172.21.15.171:6789/0 conn(0x7f2fb413c8c0 legacy=0x7f2fb4072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 >> v1:172.21.15.87:6789/0 conn(0x7f2fb4061830 legacy=0x7f2fb413c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: _finish_hunting 0 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 monclient: found mon.noname-b 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:51.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4070f90 con 0x7f2fb406ff10 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: _finish_auth 0 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: _check_auth_tickets 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2fa4001690 con 0x7f2fb406ff10 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fbc5ab640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fbc5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fb9345640 10 monclient: dump: 2024-03-25T18:19:51.068 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.065+0000 7f2fbc5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:51.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fa4001870 con 0x7f2fb406ff10 2024-03-25T18:19:51.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:51.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 1 -- 172.21.15.87:0/2965370836 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2fa4001c00 con 0x7f2fb406ff10 2024-03-25T18:19:51.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fbc5ab640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:51.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fbc5ab640 4 set_mon_vals no callback set 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.079+0000 7f2fb9345640 10 monclient: dump: 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:51.081 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:51.082 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 10 monclient: shutdown 2024-03-25T18:19:51.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:51.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/2965370836 >> v1:172.21.15.87:6790/0 conn(0x7f2fb406ff10 legacy=0x7f2fb40723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/2965370836 shutdown_connections 2024-03-25T18:19:51.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/2965370836 >> 172.21.15.87:0/2965370836 conn(0x7f2fb40b11e0 msgr2=0x7f2fb40b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/2965370836 shutdown_connections 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.080+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/2965370836 wait complete. 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 10 monclient: build_initial_monmap 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 10 monclient: monmap: 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:51.085 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 librados: starting msgr at 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 librados: starting objecter 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 Processor -- start 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 -- start start 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 librados: setting wanted keys 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 1 librados: calling monclient init 2024-03-25T18:19:51.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.081+0000 7f2fbc5ab640 10 monclient: init 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb4061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb406ff10 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb413c8c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4139440 con 0x7f2fb413c8c0 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb41372a0 con 0x7f2fb4061830 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4137580 con 0x7f2fb406ff10 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:51.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.082+0000 7f2fbc5ab640 10 monclient(hunting): authenticate will time out at 8106.377357s 2024-03-25T18:19:51.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.085+0000 7f2fb9b46640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f2fb406ff10 0x7f2fb413b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:34978/0 (socket says 172.21.15.87:34978) 2024-03-25T18:19:51.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.085+0000 7f2fb9b46640 1 -- 172.21.15.87:0/1351744768 learned_addr learned my addr 172.21.15.87:0/1351744768 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:51.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.086+0000 7f2faaffd640 10 client.?.objecter ms_handle_connect 0x7f2fb406ff10 2024-03-25T18:19:51.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.086+0000 7f2faaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:51.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.090+0000 7f2faaffd640 10 client.?.objecter ms_handle_connect 0x7f2fb4061830 2024-03-25T18:19:51.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.090+0000 7f2faaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:51.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.095+0000 7f2faaffd640 10 client.?.objecter ms_handle_connect 0x7f2fb413c8c0 2024-03-25T18:19:51.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.095+0000 7f2faaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647738371 0 0) 0x7f2fb41372a0 con 0x7f2fb4061830 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:19:51.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.099+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f98004eb0 con 0x7f2fb4061830 2024-03-25T18:19:51.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478419732 0 0) 0x7f2fb4137580 con 0x7f2fb406ff10 2024-03-25T18:19:51.103 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:19:51.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.102+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb41372a0 con 0x7f2fb406ff10 2024-03-25T18:19:51.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971444838 0 0) 0x7f2f98004eb0 con 0x7f2fb4061830 2024-03-25T18:19:51.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb4137580 con 0x7f2fb4061830 2024-03-25T18:19:51.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2fb00018b0 con 0x7f2fb4061830 2024-03-25T18:19:51.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 10 monclient(hunting): dump: 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:51.115 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:51.116 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.113+0000 7f2faaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:51.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1526869922 0 0) 0x7f2fb41372a0 con 0x7f2fb406ff10 2024-03-25T18:19:51.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f98004eb0 con 0x7f2fb406ff10 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2fa4001b80 con 0x7f2fb406ff10 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.118+0000 7f2faaffd640 10 monclient(hunting): dump: 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:51.120 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:51.121 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:51.121 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928018512 0 0) 0x7f2fb4139440 con 0x7f2fb413c8c0 2024-03-25T18:19:51.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:51.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:51.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:51.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:19:51.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.121+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb41372a0 con 0x7f2fb413c8c0 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3888934371 0 0) 0x7f2fb4137580 con 0x7f2fb4061830 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 >> v1:172.21.15.87:6790/0 conn(0x7f2fb406ff10 legacy=0x7f2fb413b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 >> v1:172.21.15.87:6789/0 conn(0x7f2fb413c8c0 legacy=0x7f2fb41382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 1 monclient: found mon.b 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4139690 con 0x7f2fb4061830 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:51.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2faaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 10 monclient: _renew_subs 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb413fc90 con 0x7f2fb4061830 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 10 monclient: _renew_subs 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb41e3d30 con 0x7f2fb4061830 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.122+0000 7f2fbc5ab640 1 librados: init done 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.123+0000 7f2fbc5ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:51.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.123+0000 7f2fbc5ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:51.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.123+0000 7f2fbc5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:51.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.123+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb40627b0 con 0x7f2fb4061830 2024-03-25T18:19:51.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb00035f0 con 0x7f2fb4061830 2024-03-25T18:19:51.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:51.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2fb0003900 con 0x7f2fb4061830 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2faaffd640 10 monclient: dump: 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:51.149 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:51.150 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:51.150 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.147+0000 7f2fb8b44640 4 set_mon_vals no callback set 2024-03-25T18:19:51.150 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.148+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2fb00198d0 con 0x7f2fb4061830 2024-03-25T18:19:51.151 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.149+0000 7f2fb9b46640 10 monclient: get_auth_request con 0x7f2f9806ded0 auth_method 0 2024-03-25T18:19:51.151 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.149+0000 7f2faaffd640 10 client.4459.objecter ms_handle_connect 0x7f2f9806ded0 2024-03-25T18:19:51.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.151+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f2fb00535e0 con 0x7f2fb4061830 2024-03-25T18:19:51.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.151+0000 7f2faaffd640 10 client.4459.objecter ms_dispatch 0x7f2fb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:51.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.151+0000 7f2faaffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:51.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.151+0000 7f2faaffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:51.152 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.151+0000 7f2faaffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:19:51.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.424+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fb004ef20 con 0x7f2fb4061830 2024-03-25T18:19:51.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.424+0000 7f2faaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:51.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.424+0000 7f2faaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:51.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.671+0000 7f2fbc5ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:51.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.671+0000 7f2fbc5ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:51.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.671+0000 7f2fbc5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:19:51.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.671+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb40727b0 con 0x7f2fb4061830 2024-03-25T18:19:51.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.718+0000 7f2faaffd640 1 -- 172.21.15.87:0/1351744768 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f2fb0051020 con 0x7f2fb4061830 2024-03-25T18:19:51.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.718+0000 7f2faaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:51.720 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.718+0000 7f2faaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:51.720 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:51.720 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:51.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 10 librados: watch_flush enter 2024-03-25T18:19:51.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 10 librados: watch_flush exit 2024-03-25T18:19:51.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:19:51.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 >> v1:172.21.15.171:6800/3417775508 conn(0x7f2f9806ded0 legacy=0x7f2f98070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 10 monclient: shutdown 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 >> v1:172.21.15.171:6789/0 conn(0x7f2fb4061830 legacy=0x7f2fb4137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 shutdown_connections 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 >> 172.21.15.87:0/1351744768 conn(0x7f2fb40b11e0 msgr2=0x7f2fb406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 shutdown_connections 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.720+0000 7f2fbc5ab640 1 -- 172.21.15.87:0/1351744768 wait complete. 2024-03-25T18:19:51.723 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:51.721+0000 7f2fbc5ab640 1 librados: shutdown 2024-03-25T18:19:51.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:58.540 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:58.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.647+0000 7faaf4754640 10 monclient: get_monmap_and_config 2024-03-25T18:19:58.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.647+0000 7faaf4754640 10 monclient: build_initial_monmap 2024-03-25T18:19:58.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.647+0000 7faaf4754640 10 monclient: monmap: 2024-03-25T18:19:58.648 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:58.648 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:19:58.648330+0000 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:19:58.648330+0000 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:19:58.649 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.648+0000 7faaf4754640 1 Processor -- start 2024-03-25T18:19:58.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.648+0000 7faaf4754640 1 -- start start 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.648+0000 7faaf4754640 10 monclient: init 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-c con 0x7faaec0a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-b con 0x7faaec13f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-a con 0x7faaec1430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec142520 con 0x7faaec1430c0 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0b28a0 con 0x7faaec0a98f0 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0b2a30 con 0x7faaec13f710 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:58.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.649+0000 7faaf4754640 10 monclient(hunting): authenticate will time out at 8113.944088s 2024-03-25T18:19:58.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.655+0000 7faaf24f0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7faaec0a98f0 0x7faaec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:59914/0 (socket says 172.21.15.87:59914) 2024-03-25T18:19:58.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.655+0000 7faaf24f0640 1 -- 172.21.15.87:0/2838176447 learned_addr learned my addr 172.21.15.87:0/2838176447 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:58.665 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.664+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13837644 0 0) 0x7faaec0b28a0 con 0x7faaec0a98f0 2024-03-25T18:19:58.665 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.665+0000 7faaf14ee640 10 monclient(hunting): _negotiate 2024-03-25T18:19:58.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.665+0000 7faaf14ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:58.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.665+0000 7faaf14ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:58.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.665+0000 7faaf14ee640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:19:58.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.665+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad8003520 con 0x7faaec0a98f0 2024-03-25T18:19:58.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443171125 0 0) 0x7faad8003520 con 0x7faaec0a98f0 2024-03-25T18:19:58.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 >> v1:172.21.15.87:6790/0 conn(0x7faaec13f710 legacy=0x7faaec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:58.680 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 >> v1:172.21.15.87:6789/0 conn(0x7faaec1430c0 legacy=0x7faaec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: _finish_hunting 0 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 monclient: found mon.noname-c 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0b44c0 con 0x7faaec0a98f0 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: _finish_auth 0 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: _check_auth_tickets 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7faae8001690 con 0x7faaec0a98f0 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf14ee640 10 monclient: dump: 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.681 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf4754640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:19:58.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.679+0000 7faaf4754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faae80017f0 con 0x7faaec0a98f0 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 1 -- 172.21.15.87:0/2838176447 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7faae8001c00 con 0x7faaec0a98f0 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf4754640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf4754640 4 set_mon_vals no callback set 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.703+0000 7faaf14ee640 10 monclient: dump: 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.704 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:58.705 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:58.705 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:58.705 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:58.705 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:58.705 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 10 monclient: shutdown 2024-03-25T18:19:58.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:58.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 1 -- 172.21.15.87:0/2838176447 >> v1:172.21.15.171:6789/0 conn(0x7faaec0a98f0 legacy=0x7faaec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:58.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 1 -- 172.21.15.87:0/2838176447 shutdown_connections 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 1 -- 172.21.15.87:0/2838176447 >> 172.21.15.87:0/2838176447 conn(0x7faaec0b7ea0 msgr2=0x7faaec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 1 -- 172.21.15.87:0/2838176447 shutdown_connections 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.704+0000 7faaf4754640 1 -- 172.21.15.87:0/2838176447 wait complete. 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 10 monclient: build_initial_monmap 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 10 monclient: monmap: 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:19:58.707 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 librados: starting msgr at 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 librados: starting objecter 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 Processor -- start 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 -- start start 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 librados: setting wanted keys 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 1 librados: calling monclient init 2024-03-25T18:19:58.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.705+0000 7faaf4754640 10 monclient: init 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-a con 0x7faaec0a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-b con 0x7faaec13f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient(hunting): picked mon.noname-c con 0x7faaec1430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec138b20 con 0x7faaec0a98f0 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec137a80 con 0x7faaec13f710 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec139880 con 0x7faaec1430c0 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:58.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.706+0000 7faaf4754640 10 monclient(hunting): authenticate will time out at 8114.001588s 2024-03-25T18:19:58.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.714+0000 7faaf24f0640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7faaec0a98f0 0x7faaec137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:57802/0 (socket says 172.21.15.87:57802) 2024-03-25T18:19:58.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.714+0000 7faaf24f0640 1 -- 172.21.15.87:0/891149203 learned_addr learned my addr 172.21.15.87:0/891149203 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:19:58.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.717+0000 7faae2ffd640 10 client.?.objecter ms_handle_connect 0x7faaec0a98f0 2024-03-25T18:19:58.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.717+0000 7faae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:58.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.723+0000 7faae2ffd640 10 client.?.objecter ms_handle_connect 0x7faaec1430c0 2024-03-25T18:19:58.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.723+0000 7faae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:58.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.725+0000 7faae2ffd640 10 client.?.objecter ms_handle_connect 0x7faaec13f710 2024-03-25T18:19:58.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.725+0000 7faae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838931070 0 0) 0x7faaec138b20 con 0x7faaec0a98f0 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:19:58.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.727+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad0004eb0 con 0x7faaec0a98f0 2024-03-25T18:19:58.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941092553 0 0) 0x7faaec139880 con 0x7faaec1430c0 2024-03-25T18:19:58.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:58.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:58.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:58.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:19:58.739 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.737+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaec138b20 con 0x7faaec1430c0 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683747318 0 0) 0x7faaec137a80 con 0x7faaec13f710 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:19:58.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.741+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaec139880 con 0x7faaec13f710 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831283007 0 0) 0x7faad0004eb0 con 0x7faaec0a98f0 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaec137a80 con 0x7faaec0a98f0 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7faae8001bc0 con 0x7faaec0a98f0 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:58.747 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.746+0000 7faae2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:58.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171232098 0 0) 0x7faaec138b20 con 0x7faaec1430c0 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faad0004eb0 con 0x7faaec1430c0 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7faae4001b10 con 0x7faaec1430c0 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.754+0000 7faae2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:58.756 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:58.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039750978 0 0) 0x7faaec137a80 con 0x7faaec0a98f0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 >> v1:172.21.15.87:6790/0 conn(0x7faaec1430c0 legacy=0x7faaec1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 >> v1:172.21.15.171:6789/0 conn(0x7faaec13f710 legacy=0x7faaec139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 1 monclient: found mon.a 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec139a10 con 0x7faaec0a98f0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 10 monclient: _renew_subs 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec1dab00 con 0x7faaec0a98f0 2024-03-25T18:19:58.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 10 monclient: _renew_subs 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec13b4f0 con 0x7faaec0a98f0 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.757+0000 7faaf4754640 1 librados: init done 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.758+0000 7faaf4754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.758+0000 7faaf4754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:58.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.758+0000 7faaf4754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:58.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:58.758+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec1dae00 con 0x7faaec0a98f0 2024-03-25T18:19:59.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faae8004a90 con 0x7faaec0a98f0 2024-03-25T18:19:59.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:59.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7faae8004da0 con 0x7faaec0a98f0 2024-03-25T18:19:59.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faae2ffd640 10 monclient: dump: 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:19:59.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.067+0000 7faaf0ced640 4 set_mon_vals no callback set 2024-03-25T18:19:59.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.068+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7faae801ad70 con 0x7faaec0a98f0 2024-03-25T18:19:59.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.069+0000 7faaf1cef640 10 monclient: get_auth_request con 0x7faad006df70 auth_method 0 2024-03-25T18:19:59.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.069+0000 7faae2ffd640 10 client.4461.objecter ms_handle_connect 0x7faad006df70 2024-03-25T18:19:59.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7faae8054920 con 0x7faaec0a98f0 2024-03-25T18:19:59.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 10 client.4461.objecter ms_dispatch 0x7faaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:59.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:59.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:59.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T18:19:59.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faae8050260 con 0x7faaec0a98f0 2024-03-25T18:19:59.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:59.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.074+0000 7faae2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:59.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.283+0000 7faaf4754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:59.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.283+0000 7faaf4754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:59.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.283+0000 7faaf4754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:19:59.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.283+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaec0b3e80 con 0x7faaec0a98f0 2024-03-25T18:19:59.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.310+0000 7faae2ffd640 1 -- 172.21.15.87:0/891149203 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7faae8052360 con 0x7faaec0a98f0 2024-03-25T18:19:59.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.310+0000 7faae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:59.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.310+0000 7faae2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:59.311 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:19:59.311 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 10 librados: watch_flush enter 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 10 librados: watch_flush exit 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 10 client.4461.objecter successfully canceled tick 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 >> v1:172.21.15.171:6800/3417775508 conn(0x7faad006df70 legacy=0x7faad0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 10 monclient: shutdown 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 >> v1:172.21.15.87:6789/0 conn(0x7faaec0a98f0 legacy=0x7faaec137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 shutdown_connections 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 >> 172.21.15.87:0/891149203 conn(0x7faaec0b7ea0 msgr2=0x7faaec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 shutdown_connections 2024-03-25T18:19:59.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.312+0000 7faaf4754640 1 -- 172.21.15.87:0/891149203 wait complete. 2024-03-25T18:19:59.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:19:59.313+0000 7faaf4754640 1 librados: shutdown 2024-03-25T18:19:59.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:06.131 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:06.246 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.245+0000 7f6adbdf6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.245+0000 7f6adbdf6640 10 monclient: build_initial_monmap 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.245+0000 7f6adbdf6640 10 monclient: monmap: 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:06.246754+0000 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:06.246754+0000 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:06.247 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.248 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.246+0000 7f6adbdf6640 1 Processor -- start 2024-03-25T18:20:06.248 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.246+0000 7f6adbdf6640 1 -- start start 2024-03-25T18:20:06.248 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.246+0000 7f6adbdf6640 10 monclient: init 2024-03-25T18:20:06.248 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad40a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad41430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad4142520 con 0x7f6ad40a98f0 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40b28a0 con 0x7f6ad413f710 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40b2a30 con 0x7f6ad41430c0 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:06.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.247+0000 7f6adbdf6640 10 monclient(hunting): authenticate will time out at 8121.542294s 2024-03-25T18:20:06.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.250+0000 7f6ad9b92640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f6ad40a98f0 0x7f6ad40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:34274/0 (socket says 172.21.15.87:34274) 2024-03-25T18:20:06.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.250+0000 7f6ad9b92640 1 -- 172.21.15.87:0/2543919654 learned_addr learned my addr 172.21.15.87:0/2543919654 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029584547 0 0) 0x7f6ad4142520 con 0x7f6ad40a98f0 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 10 monclient(hunting): _negotiate 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:20:06.256 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.255+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac0003520 con 0x7f6ad40a98f0 2024-03-25T18:20:06.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342190339 0 0) 0x7f6ac0003520 con 0x7f6ad40a98f0 2024-03-25T18:20:06.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 >> v1:172.21.15.87:6790/0 conn(0x7f6ad41430c0 legacy=0x7f6ad414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:06.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 >> v1:172.21.15.171:6789/0 conn(0x7f6ad413f710 legacy=0x7f6ad41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: _finish_hunting 0 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 monclient: found mon.noname-a 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad40b44c0 con 0x7f6ad40a98f0 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: _finish_auth 0 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: _check_auth_tickets 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6ad0001690 con 0x7f6ad40a98f0 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6ad8b90640 10 monclient: dump: 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:06.266 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6adbdf6640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:20:06.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.264+0000 7f6adbdf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ad0001860 con 0x7f6ad40a98f0 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 1 -- 172.21.15.87:0/2543919654 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6ad0001cb0 con 0x7f6ad40a98f0 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:06.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6ad8b90640 10 monclient: dump: 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6adbdf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:06.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.274+0000 7f6adbdf6640 4 set_mon_vals no callback set 2024-03-25T18:20:06.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 10 monclient: shutdown 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 1 -- 172.21.15.87:0/2543919654 >> v1:172.21.15.87:6789/0 conn(0x7f6ad40a98f0 legacy=0x7f6ad40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 1 -- 172.21.15.87:0/2543919654 shutdown_connections 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 1 -- 172.21.15.87:0/2543919654 >> 172.21.15.87:0/2543919654 conn(0x7f6ad40b7ea0 msgr2=0x7f6ad40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 1 -- 172.21.15.87:0/2543919654 shutdown_connections 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.275+0000 7f6adbdf6640 1 -- 172.21.15.87:0/2543919654 wait complete. 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.276+0000 7f6adbdf6640 10 monclient: build_initial_monmap 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.276+0000 7f6adbdf6640 10 monclient: monmap: 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:06.278 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.276+0000 7f6adbdf6640 1 librados: starting msgr at 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.276+0000 7f6adbdf6640 1 librados: starting objecter 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.276+0000 7f6adbdf6640 1 Processor -- start 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 -- start start 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 librados: setting wanted keys 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 librados: calling monclient init 2024-03-25T18:20:06.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient: init 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40abb10 con 0x7f6ad41430c0 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad41d7850 con 0x7f6ad413f710 2024-03-25T18:20:06.280 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad41d7500 con 0x7f6ad40a98f0 2024-03-25T18:20:06.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:06.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.277+0000 7f6adbdf6640 10 monclient(hunting): authenticate will time out at 8121.572506s 2024-03-25T18:20:06.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.283+0000 7f6ad9391640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6ad413f710 0x7f6ad41d7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:36298/0 (socket says 172.21.15.87:36298) 2024-03-25T18:20:06.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.284+0000 7f6ad9391640 1 -- 172.21.15.87:0/3242077957 learned_addr learned my addr 172.21.15.87:0/3242077957 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:06.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.286+0000 7f6aca7fc640 10 client.?.objecter ms_handle_connect 0x7f6ad413f710 2024-03-25T18:20:06.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.286+0000 7f6aca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:06.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.288+0000 7f6aca7fc640 10 client.?.objecter ms_handle_connect 0x7f6ad41430c0 2024-03-25T18:20:06.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.288+0000 7f6aca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:06.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680694120 0 0) 0x7f6ad40abb10 con 0x7f6ad41430c0 2024-03-25T18:20:06.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:06.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:06.294 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:06.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:20:06.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.293+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8004eb0 con 0x7f6ad41430c0 2024-03-25T18:20:06.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115532282 0 0) 0x7f6ad41d7850 con 0x7f6ad413f710 2024-03-25T18:20:06.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:06.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:06.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:06.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:20:06.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.298+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad40abb10 con 0x7f6ad413f710 2024-03-25T18:20:06.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.302+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857866109 0 0) 0x7f6ab8004eb0 con 0x7f6ad41430c0 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ad41d7850 con 0x7f6ad41430c0 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6acc001ad0 con 0x7f6ad41430c0 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:06.304 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.305 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.303+0000 7f6aca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:06.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.307+0000 7f6aca7fc640 10 client.?.objecter ms_handle_connect 0x7f6ad40a98f0 2024-03-25T18:20:06.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.307+0000 7f6aca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137441213 0 0) 0x7f6ad40abb10 con 0x7f6ad413f710 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ab8004eb0 con 0x7f6ad413f710 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6ac4001920 con 0x7f6ad413f710 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.308+0000 7f6aca7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.309 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:06.310 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:06.310 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:06.310 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025367731 0 0) 0x7f6ad41d7850 con 0x7f6ad41430c0 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 >> v1:172.21.15.87:6790/0 conn(0x7f6ad40a98f0 legacy=0x7f6ad41384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 >> v1:172.21.15.171:6789/0 conn(0x7f6ad413f710 legacy=0x7f6ad41d7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 1 monclient: found mon.a 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4137750 con 0x7f6ad41430c0 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:06.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6aca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 5 monclient: authenticate success, global_id 4467 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 10 monclient: _renew_subs 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad4137a00 con 0x7f6ad41430c0 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 10 monclient: _renew_subs 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad4134800 con 0x7f6ad41430c0 2024-03-25T18:20:06.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.315+0000 7f6adbdf6640 1 librados: init done 2024-03-25T18:20:06.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.316+0000 7f6adbdf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:06.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.316+0000 7f6adbdf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:06.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.316+0000 7f6adbdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.316+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad41d7690 con 0x7f6ad41430c0 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6acc001fe0 con 0x7f6ad41430c0 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6acc003b10 con 0x7f6ad41430c0 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 10 monclient: dump: 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:06.605 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:06.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6acbfff640 4 set_mon_vals no callback set 2024-03-25T18:20:06.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.604+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f6acc019ae0 con 0x7f6ad41430c0 2024-03-25T18:20:06.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.605+0000 7f6ad9b92640 10 monclient: get_auth_request con 0x7f6ab806ddf0 auth_method 0 2024-03-25T18:20:06.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.605+0000 7f6aca7fc640 10 client.4467.objecter ms_handle_connect 0x7f6ab806ddf0 2024-03-25T18:20:06.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.610+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f6acc0537d0 con 0x7f6ad41430c0 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.610+0000 7f6aca7fc640 10 client.4467.objecter ms_dispatch 0x7f6ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.610+0000 7f6aca7fc640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.610+0000 7f6aca7fc640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.610+0000 7f6aca7fc640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.611+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6acc04f110 con 0x7f6ad41430c0 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.611+0000 7f6aca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:06.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.611+0000 7f6aca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:06.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.843+0000 7f6adbdf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:06.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.843+0000 7f6adbdf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:06.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.843+0000 7f6adbdf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:06.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.843+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ad41d79e0 con 0x7f6ad41430c0 2024-03-25T18:20:06.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.882+0000 7f6aca7fc640 1 -- 172.21.15.87:0/3242077957 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f6acc051210 con 0x7f6ad41430c0 2024-03-25T18:20:06.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.882+0000 7f6aca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:06.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.882+0000 7f6aca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:06.883 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:06.883 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 10 librados: watch_flush enter 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 10 librados: watch_flush exit 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 10 client.4467.objecter successfully canceled tick 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 >> v1:172.21.15.171:6800/3417775508 conn(0x7f6ab806ddf0 legacy=0x7f6ab80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 10 monclient: shutdown 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 >> v1:172.21.15.87:6789/0 conn(0x7f6ad41430c0 legacy=0x7f6ad4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.884+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 shutdown_connections 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.885+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 >> 172.21.15.87:0/3242077957 conn(0x7f6ad40b7ea0 msgr2=0x7f6ad413ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.885+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 shutdown_connections 2024-03-25T18:20:06.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.885+0000 7f6adbdf6640 1 -- 172.21.15.87:0/3242077957 wait complete. 2024-03-25T18:20:06.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:06.885+0000 7f6adbdf6640 1 librados: shutdown 2024-03-25T18:20:06.897 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:07.353 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:55.067 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.360 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:55.073 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.386 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:05:55.100 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.414 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:55.127 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.416 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:55.094 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.418 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:55.131 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.422 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:55.101 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.451 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:05:55.130 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:07.461 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:05:55.174 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:13.705 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:13.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.817+0000 7f2c84fe2640 10 monclient: get_monmap_and_config 2024-03-25T18:20:13.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.817+0000 7f2c84fe2640 10 monclient: build_initial_monmap 2024-03-25T18:20:13.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.817+0000 7f2c84fe2640 10 monclient: monmap: 2024-03-25T18:20:13.819 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:13.819227+0000 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:13.819227+0000 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:13.820 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:13.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 1 Processor -- start 2024-03-25T18:20:13.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 1 -- start start 2024-03-25T18:20:13.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient: init 2024-03-25T18:20:13.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:13.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-a con 0x7f2c800a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-c con 0x7f2c8013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-b con 0x7f2c801430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c80142520 con 0x7f2c800a98f0 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c800b28a0 con 0x7f2c8013f710 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c800b2a30 con 0x7f2c801430c0 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.818+0000 7f2c84fe2640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:13.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.819+0000 7f2c84fe2640 10 monclient(hunting): authenticate will time out at 8129.114844s 2024-03-25T18:20:13.829 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.827+0000 7f2c75d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2c8013f710 0x7f2c801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:56462/0 (socket says 172.21.15.87:56462) 2024-03-25T18:20:13.830 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.828+0000 7f2c75d9b640 1 -- 172.21.15.87:0/1926345367 learned_addr learned my addr 172.21.15.87:0/1926345367 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:13.840 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306609020 0 0) 0x7f2c800b28a0 con 0x7f2c8013f710 2024-03-25T18:20:13.840 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:13.840 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:13.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:13.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:20:13.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.838+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c6c003520 con 0x7f2c8013f710 2024-03-25T18:20:13.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144316936 0 0) 0x7f2c80142520 con 0x7f2c800a98f0 2024-03-25T18:20:13.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:13.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:13.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:13.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:20:13.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.850+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c800b28a0 con 0x7f2c800a98f0 2024-03-25T18:20:13.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711874666 0 0) 0x7f2c800b2a30 con 0x7f2c801430c0 2024-03-25T18:20:13.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:13.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:13.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:13.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:20:13.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.856+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c80142520 con 0x7f2c801430c0 2024-03-25T18:20:13.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5886319 0 0) 0x7f2c6c003520 con 0x7f2c8013f710 2024-03-25T18:20:13.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 >> v1:172.21.15.87:6790/0 conn(0x7f2c801430c0 legacy=0x7f2c8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:13.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 >> v1:172.21.15.87:6789/0 conn(0x7f2c800a98f0 legacy=0x7f2c800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:13.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:13.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c800b44c0 con 0x7f2c8013f710 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2c68001690 con 0x7f2c8013f710 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c84fe2640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c84fe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:13.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c7dd9b640 10 monclient: dump: 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:13.866 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:13.867 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.861+0000 7f2c84fe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:13.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c680017f0 con 0x7f2c8013f710 2024-03-25T18:20:13.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:13.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 1 -- 172.21.15.87:0/1926345367 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2c68001c00 con 0x7f2c8013f710 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c84fe2640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c84fe2640 4 set_mon_vals no callback set 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.875+0000 7f2c7dd9b640 10 monclient: dump: 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:13.878 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:13.879 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:13.879 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:13.879 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:13.879 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 10 monclient: shutdown 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 1 -- 172.21.15.87:0/1926345367 >> v1:172.21.15.171:6789/0 conn(0x7f2c8013f710 legacy=0x7f2c801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 1 -- 172.21.15.87:0/1926345367 shutdown_connections 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 1 -- 172.21.15.87:0/1926345367 >> 172.21.15.87:0/1926345367 conn(0x7f2c800b7ea0 msgr2=0x7f2c800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 1 -- 172.21.15.87:0/1926345367 shutdown_connections 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.876+0000 7f2c84fe2640 1 -- 172.21.15.87:0/1926345367 wait complete. 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.877+0000 7f2c84fe2640 10 monclient: build_initial_monmap 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.877+0000 7f2c84fe2640 10 monclient: monmap: 2024-03-25T18:20:13.882 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.877+0000 7f2c84fe2640 1 librados: starting msgr at 2024-03-25T18:20:13.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.877+0000 7f2c84fe2640 1 librados: starting objecter 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 1 Processor -- start 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 1 -- start start 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 1 librados: setting wanted keys 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 1 librados: calling monclient init 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 10 monclient: init 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.878+0000 7f2c84fe2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-c con 0x7f2c800a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 10 monclient(hunting): picked mon.noname-a con 0x7f2c801430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:13.884 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c80139070 con 0x7f2c801430c0 2024-03-25T18:20:13.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c80133710 con 0x7f2c8013f710 2024-03-25T18:20:13.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c801338a0 con 0x7f2c800a98f0 2024-03-25T18:20:13.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:13.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.879+0000 7f2c84fe2640 10 monclient(hunting): authenticate will time out at 8129.175034s 2024-03-25T18:20:13.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.884+0000 7f2c7ed9d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f2c801430c0 0x7f2c80133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:60914/0 (socket says 172.21.15.87:60914) 2024-03-25T18:20:13.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.884+0000 7f2c7ed9d640 1 -- 172.21.15.87:0/2913844747 learned_addr learned my addr 172.21.15.87:0/2913844747 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:13.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.886+0000 7f2c777fe640 10 client.?.objecter ms_handle_connect 0x7f2c801430c0 2024-03-25T18:20:13.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.886+0000 7f2c777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:13.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987859009 0 0) 0x7f2c80139070 con 0x7f2c801430c0 2024-03-25T18:20:13.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:13.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:13.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:13.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:20:13.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.891+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60004eb0 con 0x7f2c801430c0 2024-03-25T18:20:13.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.892+0000 7f2c777fe640 10 client.?.objecter ms_handle_connect 0x7f2c800a98f0 2024-03-25T18:20:13.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.892+0000 7f2c777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:13.900 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.898+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349866422 0 0) 0x7f2c60004eb0 con 0x7f2c801430c0 2024-03-25T18:20:13.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.898+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c80139070 con 0x7f2c801430c0 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934311115 0 0) 0x7f2c801338a0 con 0x7f2c800a98f0 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:20:13.903 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.901+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60004eb0 con 0x7f2c800a98f0 2024-03-25T18:20:13.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.903+0000 7f2c777fe640 10 client.?.objecter ms_handle_connect 0x7f2c8013f710 2024-03-25T18:20:13.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.903+0000 7f2c777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2c78001db0 con 0x7f2c801430c0 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient(hunting): dump: 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:13.920 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1920862147 0 0) 0x7f2c80139070 con 0x7f2c801430c0 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 >> v1:172.21.15.87:6790/0 conn(0x7f2c800a98f0 legacy=0x7f2c801378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 >> v1:172.21.15.171:6789/0 conn(0x7f2c8013f710 legacy=0x7f2c801daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 monclient: found mon.a 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:13.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c80133a30 con 0x7f2c801430c0 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 10 monclient: _renew_subs 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c800b7100 con 0x7f2c801430c0 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:13.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 10 monclient: _renew_subs 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c80136e60 con 0x7f2c801430c0 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.916+0000 7f2c84fe2640 1 librados: init done 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.917+0000 7f2c84fe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.917+0000 7f2c84fe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.917+0000 7f2c84fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:13.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:13.917+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c800627b0 con 0x7f2c801430c0 2024-03-25T18:20:14.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c78001f10 con 0x7f2c801430c0 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2c78003de0 con 0x7f2c801430c0 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c777fe640 10 monclient: dump: 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:14.219 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.216+0000 7f2c7d59a640 4 set_mon_vals no callback set 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.217+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2c78004000 con 0x7f2c801430c0 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.218+0000 7f2c7e59c640 10 monclient: get_auth_request con 0x7f2c6006db90 auth_method 0 2024-03-25T18:20:14.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.218+0000 7f2c777fe640 10 client.4473.objecter ms_handle_connect 0x7f2c6006db90 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.224+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f2c78054ac0 con 0x7f2c801430c0 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.224+0000 7f2c777fe640 10 client.4473.objecter ms_dispatch 0x7f2c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.224+0000 7f2c777fe640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.224+0000 7f2c777fe640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.225+0000 7f2c777fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.225+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c780505d0 con 0x7f2c801430c0 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.225+0000 7f2c777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:14.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.225+0000 7f2c777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:14.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.441+0000 7f2c84fe2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:14.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.441+0000 7f2c84fe2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:14.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.441+0000 7f2c84fe2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:14.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.441+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c801374d0 con 0x7f2c801430c0 2024-03-25T18:20:14.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.473+0000 7f2c777fe640 1 -- 172.21.15.87:0/2913844747 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f2c78052500 con 0x7f2c801430c0 2024-03-25T18:20:14.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.473+0000 7f2c777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:14.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.473+0000 7f2c777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:14.476 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:14.476 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 10 librados: watch_flush enter 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 10 librados: watch_flush exit 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 >> v1:172.21.15.171:6800/3417775508 conn(0x7f2c6006db90 legacy=0x7f2c60070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 10 monclient: shutdown 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 >> v1:172.21.15.87:6789/0 conn(0x7f2c801430c0 legacy=0x7f2c80133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 shutdown_connections 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 >> 172.21.15.87:0/2913844747 conn(0x7f2c800b7ea0 msgr2=0x7f2c80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 shutdown_connections 2024-03-25T18:20:14.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 -- 172.21.15.87:0/2913844747 wait complete. 2024-03-25T18:20:14.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:14.476+0000 7f2c84fe2640 1 librados: shutdown 2024-03-25T18:20:14.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:21.297 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.412+0000 7f733eccf640 10 monclient: get_monmap_and_config 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.412+0000 7f733eccf640 10 monclient: build_initial_monmap 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.412+0000 7f733eccf640 10 monclient: monmap: 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:21.413182+0000 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:21.413182+0000 2024-03-25T18:20:21.413 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:21.414 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:21.414 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:21.414 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:21.414 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:21.414 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 1 Processor -- start 2024-03-25T18:20:21.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 1 -- start start 2024-03-25T18:20:21.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient: init 2024-03-25T18:20:21.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:21.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-b con 0x7f73380a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-c con 0x7f733813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-a con 0x7f73381430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.413+0000 7f733eccf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7338142520 con 0x7f73381430c0 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.414+0000 7f733eccf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73380b28a0 con 0x7f733813f710 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.414+0000 7f733eccf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73380b2a30 con 0x7f73380a98f0 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.414+0000 7f733eccf640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:21.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.414+0000 7f733eccf640 10 monclient(hunting): authenticate will time out at 8136.708868s 2024-03-25T18:20:21.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.423+0000 7f733d26c640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f73381430c0 0x7f733814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:40636/0 (socket says 172.21.15.87:40636) 2024-03-25T18:20:21.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.423+0000 7f733d26c640 1 -- 172.21.15.87:0/3154270037 learned_addr learned my addr 172.21.15.87:0/3154270037 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:21.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.438+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784897417 0 0) 0x7f73380b28a0 con 0x7f733813f710 2024-03-25T18:20:21.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.439+0000 7f732f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.439+0000 7f732f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.439+0000 7f732f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.439+0000 7f732f7fe640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:20:21.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.439+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7320003520 con 0x7f733813f710 2024-03-25T18:20:21.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957996458 0 0) 0x7f7338142520 con 0x7f73381430c0 2024-03-25T18:20:21.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:20:21.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.443+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73380b28a0 con 0x7f73381430c0 2024-03-25T18:20:21.449 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447231509 0 0) 0x7f73380b2a30 con 0x7f73380a98f0 2024-03-25T18:20:21.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:20:21.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.448+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338142520 con 0x7f73380a98f0 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552241243 0 0) 0x7f73380b28a0 con 0x7f73381430c0 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 >> v1:172.21.15.87:6790/0 conn(0x7f73380a98f0 legacy=0x7f73380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 >> v1:172.21.15.171:6789/0 conn(0x7f733813f710 legacy=0x7f73381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:20:21.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73380b44c0 con 0x7f73381430c0 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7330001670 con 0x7f73381430c0 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.457+0000 7f732f7fe640 10 monclient: dump: 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:21.460 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.458+0000 7f733eccf640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:20:21.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.458+0000 7f733eccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73300017d0 con 0x7f73381430c0 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 1 -- 172.21.15.87:0/3154270037 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7330001be0 con 0x7f73381430c0 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f733eccf640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f733eccf640 4 set_mon_vals no callback set 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.477+0000 7f732f7fe640 10 monclient: dump: 2024-03-25T18:20:21.478 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:21.479 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 10 monclient: shutdown 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 1 -- 172.21.15.87:0/3154270037 >> v1:172.21.15.87:6789/0 conn(0x7f73381430c0 legacy=0x7f733814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 1 -- 172.21.15.87:0/3154270037 shutdown_connections 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 1 -- 172.21.15.87:0/3154270037 >> 172.21.15.87:0/3154270037 conn(0x7f73380b7ea0 msgr2=0x7f73380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 1 -- 172.21.15.87:0/3154270037 shutdown_connections 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.478+0000 7f733eccf640 1 -- 172.21.15.87:0/3154270037 wait complete. 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 10 monclient: build_initial_monmap 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 10 monclient: monmap: 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:21.481 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 librados: starting msgr at 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 librados: starting objecter 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 Processor -- start 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 -- start start 2024-03-25T18:20:21.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 librados: setting wanted keys 2024-03-25T18:20:21.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 1 librados: calling monclient init 2024-03-25T18:20:21.483 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.479+0000 7f733eccf640 10 monclient: init 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-c con 0x7f73380a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-b con 0x7f733813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 10 monclient(hunting): picked mon.noname-a con 0x7f73381430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:21.484 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7338138bd0 con 0x7f73381430c0 2024-03-25T18:20:21.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.480+0000 7f733eccf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73380aba30 con 0x7f733813f710 2024-03-25T18:20:21.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.481+0000 7f733eccf640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7338137740 con 0x7f73380a98f0 2024-03-25T18:20:21.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.481+0000 7f733eccf640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:21.485 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.481+0000 7f733eccf640 10 monclient(hunting): authenticate will time out at 8136.775932s 2024-03-25T18:20:21.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.489+0000 7f732ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f733813f710 0x7f7338134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:54868/0 (socket says 172.21.15.87:54868) 2024-03-25T18:20:21.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.489+0000 7f732ffff640 1 -- 172.21.15.87:0/2289580673 learned_addr learned my addr 172.21.15.87:0/2289580673 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:21.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.490+0000 7f732d7fa640 10 client.?.objecter ms_handle_connect 0x7f733813f710 2024-03-25T18:20:21.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.490+0000 7f732d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:21.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.494+0000 7f732d7fa640 10 client.?.objecter ms_handle_connect 0x7f73380a98f0 2024-03-25T18:20:21.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.494+0000 7f732d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:21.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220827927 0 0) 0x7f73380aba30 con 0x7f733813f710 2024-03-25T18:20:21.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.505 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.505 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.505 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:20:21.505 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.503+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7324004eb0 con 0x7f733813f710 2024-03-25T18:20:21.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.511+0000 7f732d7fa640 10 client.?.objecter ms_handle_connect 0x7f73381430c0 2024-03-25T18:20:21.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.511+0000 7f732d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:21.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3351845972 0 0) 0x7f7324004eb0 con 0x7f733813f710 2024-03-25T18:20:21.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73380aba30 con 0x7f733813f710 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7328001990 con 0x7f733813f710 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 10 monclient(hunting): dump: 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:21.514 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:21.515 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:21.515 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.512+0000 7f732d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:21.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858855808 0 0) 0x7f7338137740 con 0x7f73380a98f0 2024-03-25T18:20:21.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:20:21.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.516+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7324004eb0 con 0x7f73380a98f0 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507365587 0 0) 0x7f7338138bd0 con 0x7f73381430c0 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:20:21.523 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.522+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338137740 con 0x7f73381430c0 2024-03-25T18:20:21.531 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 396075489 0 0) 0x7f73380aba30 con 0x7f733813f710 2024-03-25T18:20:21.531 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 >> v1:172.21.15.87:6790/0 conn(0x7f73380a98f0 legacy=0x7f73381384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:21.531 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 >> v1:172.21.15.87:6789/0 conn(0x7f73381430c0 legacy=0x7f73381d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:21.531 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 1 monclient: found mon.b 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7338134de0 con 0x7f733813f710 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f732d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 10 monclient: _renew_subs 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73380b1620 con 0x7f733813f710 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:21.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 10 monclient: _renew_subs 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7338134900 con 0x7f733813f710 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.530+0000 7f733eccf640 1 librados: init done 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.531+0000 7f733eccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.531+0000 7f733eccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.531+0000 7f733eccf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:21.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.531+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73380627b0 con 0x7f733813f710 2024-03-25T18:20:21.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7328001c50 con 0x7f733813f710 2024-03-25T18:20:21.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:21.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7328003a60 con 0x7f733813f710 2024-03-25T18:20:21.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732d7fa640 10 monclient: dump: 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:21.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.814+0000 7f732effd640 4 set_mon_vals no callback set 2024-03-25T18:20:21.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.815+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f7328003c80 con 0x7f733813f710 2024-03-25T18:20:21.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.815+0000 7f733ca6b640 10 monclient: get_auth_request con 0x7f732406de10 auth_method 0 2024-03-25T18:20:21.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.816+0000 7f732d7fa640 10 client.4480.objecter ms_handle_connect 0x7f732406de10 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f7328054860 con 0x7f733813f710 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 10 client.4480.objecter ms_dispatch 0x7f7338000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.822+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7328050370 con 0x7f733813f710 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.823+0000 7f732d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:21.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:21.823+0000 7f732d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:22.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.051+0000 7f733eccf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:22.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.051+0000 7f733eccf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:22.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.051+0000 7f733eccf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:22.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.051+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73381d7360 con 0x7f733813f710 2024-03-25T18:20:22.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.091+0000 7f732d7fa640 1 -- 172.21.15.87:0/2289580673 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f73280522a0 con 0x7f733813f710 2024-03-25T18:20:22.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.091+0000 7f732d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:22.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.091+0000 7f732d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:22.093 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:22.093 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:22.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 10 librados: watch_flush enter 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 10 librados: watch_flush exit 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 >> v1:172.21.15.171:6800/3417775508 conn(0x7f732406de10 legacy=0x7f73240702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 10 monclient: shutdown 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 >> v1:172.21.15.171:6789/0 conn(0x7f733813f710 legacy=0x7f7338134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 shutdown_connections 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 >> 172.21.15.87:0/2289580673 conn(0x7f73380b7ea0 msgr2=0x7f7338074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 shutdown_connections 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 -- 172.21.15.87:0/2289580673 wait complete. 2024-03-25T18:20:22.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:22.094+0000 7f733eccf640 1 librados: shutdown 2024-03-25T18:20:22.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:27.413 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:15.126 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.421 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:15.134 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.460 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:15.174 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.466 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:15.144 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.473 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:15.151 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.477 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:15.190 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.482 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:15.195 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.505 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:15.218 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:27.511 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:15.189 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:28.914 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.028+0000 7fae4d3c4640 10 monclient: get_monmap_and_config 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.028+0000 7fae4d3c4640 10 monclient: build_initial_monmap 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.028+0000 7fae4d3c4640 10 monclient: monmap: 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:29.029040+0000 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:29.029040+0000 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:29.029 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:29.030 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:29.030 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:29.030 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.029+0000 7fae4d3c4640 1 Processor -- start 2024-03-25T18:20:29.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.029+0000 7fae4d3c4640 1 -- start start 2024-03-25T18:20:29.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.029+0000 7fae4d3c4640 10 monclient: init 2024-03-25T18:20:29.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-c con 0x7fae480a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fae4813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fae481430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae48142520 con 0x7fae481430c0 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae480b28a0 con 0x7fae480a98f0 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae480b2a30 con 0x7fae4813f710 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:29.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.030+0000 7fae4d3c4640 10 monclient(hunting): authenticate will time out at 8144.325355s 2024-03-25T18:20:29.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.038+0000 7fae46ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fae480a98f0 0x7fae480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:40774/0 (socket says 172.21.15.87:40774) 2024-03-25T18:20:29.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.038+0000 7fae46ffd640 1 -- 172.21.15.87:0/4141026461 learned_addr learned my addr 172.21.15.87:0/4141026461 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073951308 0 0) 0x7fae480b28a0 con 0x7fae480a98f0 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:20:29.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.047+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae34003520 con 0x7fae480a98f0 2024-03-25T18:20:29.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505145178 0 0) 0x7fae480b2a30 con 0x7fae4813f710 2024-03-25T18:20:29.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:29.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:29.058 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:29.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:20:29.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.057+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae480b28a0 con 0x7fae4813f710 2024-03-25T18:20:29.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336483864 0 0) 0x7fae34003520 con 0x7fae480a98f0 2024-03-25T18:20:29.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 >> v1:172.21.15.87:6790/0 conn(0x7fae4813f710 legacy=0x7fae481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 >> v1:172.21.15.87:6789/0 conn(0x7fae481430c0 legacy=0x7fae4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:29.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 monclient: found mon.noname-c 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae480b44c0 con 0x7fae480a98f0 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fae38001a10 con 0x7fae480a98f0 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae467fc640 10 monclient: dump: 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.064 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae4d3c4640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:20:29.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.062+0000 7fae4d3c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae38001b70 con 0x7fae480a98f0 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 1 -- 172.21.15.87:0/4141026461 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fae38001f80 con 0x7fae480a98f0 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae467fc640 10 monclient: dump: 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:29.072 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:29.073 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:29.073 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae4d3c4640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:29.073 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.070+0000 7fae4d3c4640 4 set_mon_vals no callback set 2024-03-25T18:20:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 10 monclient: shutdown 2024-03-25T18:20:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 1 -- 172.21.15.87:0/4141026461 >> v1:172.21.15.171:6789/0 conn(0x7fae480a98f0 legacy=0x7fae480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 1 -- 172.21.15.87:0/4141026461 shutdown_connections 2024-03-25T18:20:29.075 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 1 -- 172.21.15.87:0/4141026461 >> 172.21.15.87:0/4141026461 conn(0x7fae480b7ea0 msgr2=0x7fae480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 1 -- 172.21.15.87:0/4141026461 shutdown_connections 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.071+0000 7fae4d3c4640 1 -- 172.21.15.87:0/4141026461 wait complete. 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 10 monclient: build_initial_monmap 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 10 monclient: monmap: 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.076 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 librados: starting msgr at 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 librados: starting objecter 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 Processor -- start 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 -- start start 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 librados: setting wanted keys 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 1 librados: calling monclient init 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.072+0000 7fae4d3c4640 10 monclient: init 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fae480a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-c con 0x7fae4813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fae481430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae48135730 con 0x7fae481430c0 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae481358c0 con 0x7fae480a98f0 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae48135a50 con 0x7fae4813f710 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:29.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.073+0000 7fae4d3c4640 10 monclient(hunting): authenticate will time out at 8144.368368s 2024-03-25T18:20:29.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.085+0000 7fae477fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fae481430c0 0x7fae481d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:44360/0 (socket says 172.21.15.87:44360) 2024-03-25T18:20:29.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.086+0000 7fae477fe640 1 -- 172.21.15.87:0/2180226831 learned_addr learned my addr 172.21.15.87:0/2180226831 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:29.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.087+0000 7fae3f7fe640 10 client.?.objecter ms_handle_connect 0x7fae481430c0 2024-03-25T18:20:29.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.087+0000 7fae3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:29.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.091+0000 7fae3f7fe640 10 client.?.objecter ms_handle_connect 0x7fae480a98f0 2024-03-25T18:20:29.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.091+0000 7fae3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:29.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.097+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072416145 0 0) 0x7fae481358c0 con 0x7fae480a98f0 2024-03-25T18:20:29.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.097+0000 7fae3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:29.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.097+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:29.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.097+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:29.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.098+0000 7fae3f7fe640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:20:29.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.098+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28004eb0 con 0x7fae480a98f0 2024-03-25T18:20:29.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.099+0000 7fae3f7fe640 10 client.?.objecter ms_handle_connect 0x7fae4813f710 2024-03-25T18:20:29.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.099+0000 7fae3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:29.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267334879 0 0) 0x7fae48135a50 con 0x7fae4813f710 2024-03-25T18:20:29.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:29.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:29.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:29.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.114+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae481358c0 con 0x7fae4813f710 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195314180 0 0) 0x7fae48135730 con 0x7fae481430c0 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:20:29.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.115+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48135a50 con 0x7fae481430c0 2024-03-25T18:20:29.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1520729038 0 0) 0x7fae28004eb0 con 0x7fae480a98f0 2024-03-25T18:20:29.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae48135730 con 0x7fae480a98f0 2024-03-25T18:20:29.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fae38001f40 con 0x7fae480a98f0 2024-03-25T18:20:29.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:29.118 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.116+0000 7fae3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2786493018 0 0) 0x7fae48135730 con 0x7fae480a98f0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 >> v1:172.21.15.87:6790/0 conn(0x7fae4813f710 legacy=0x7fae481d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 >> v1:172.21.15.87:6789/0 conn(0x7fae481430c0 legacy=0x7fae481d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 1 monclient: found mon.b 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae481d5000 con 0x7fae480a98f0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:29.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 10 monclient: _renew_subs 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae481d52b0 con 0x7fae480a98f0 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 10 monclient: _renew_subs 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4813c900 con 0x7fae480a98f0 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.130+0000 7fae4d3c4640 1 librados: init done 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.131+0000 7fae4d3c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:29.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.131+0000 7fae4d3c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:29.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.131+0000 7fae4d3c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.131+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4813cbb0 con 0x7fae480a98f0 2024-03-25T18:20:29.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae38004e10 con 0x7fae480a98f0 2024-03-25T18:20:29.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:29.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fae380015d0 con 0x7fae480a98f0 2024-03-25T18:20:29.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:29.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae3f7fe640 10 monclient: dump: 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:29.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.402+0000 7fae45ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:29.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.403+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fae3801ae30 con 0x7fae480a98f0 2024-03-25T18:20:29.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.403+0000 7fae3ffff640 10 monclient: get_auth_request con 0x7fae2806dda0 auth_method 0 2024-03-25T18:20:29.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.403+0000 7fae3f7fe640 10 client.4486.objecter ms_handle_connect 0x7fae2806dda0 2024-03-25T18:20:29.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.407+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fae38054ae0 con 0x7fae480a98f0 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 10 client.4486.objecter ms_dispatch 0x7fae48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae38050420 con 0x7fae480a98f0 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:29.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.408+0000 7fae3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:29.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.656+0000 7fae4d3c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:29.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.656+0000 7fae4d3c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:29.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.656+0000 7fae4d3c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:29.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.656+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae481371c0 con 0x7fae480a98f0 2024-03-25T18:20:29.780 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.779+0000 7fae3f7fe640 1 -- 172.21.15.87:0/2180226831 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fae38052520 con 0x7fae480a98f0 2024-03-25T18:20:29.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.779+0000 7fae3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:29.781 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.779+0000 7fae3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:29.781 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:29.781 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 10 librados: watch_flush enter 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 10 librados: watch_flush exit 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 10 client.4486.objecter successfully canceled tick 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 >> v1:172.21.15.171:6800/3417775508 conn(0x7fae2806dda0 legacy=0x7fae28070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 10 monclient: shutdown 2024-03-25T18:20:29.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 >> v1:172.21.15.171:6789/0 conn(0x7fae480a98f0 legacy=0x7fae481dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 shutdown_connections 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 >> 172.21.15.87:0/2180226831 conn(0x7fae480b7ea0 msgr2=0x7fae4814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.781+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 shutdown_connections 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.782+0000 7fae4d3c4640 1 -- 172.21.15.87:0/2180226831 wait complete. 2024-03-25T18:20:29.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:29.782+0000 7fae4d3c4640 1 librados: shutdown 2024-03-25T18:20:29.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:36.603 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:36.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.714+0000 7fb0ef90e640 10 monclient: get_monmap_and_config 2024-03-25T18:20:36.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.714+0000 7fb0ef90e640 10 monclient: build_initial_monmap 2024-03-25T18:20:36.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.714+0000 7fb0ef90e640 10 monclient: monmap: 2024-03-25T18:20:36.715 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:36.715586+0000 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:36.715586+0000 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:36.716 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.715+0000 7fb0ef90e640 1 Processor -- start 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.715+0000 7fb0ef90e640 1 -- start start 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.715+0000 7fb0ef90e640 10 monclient: init 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e80b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e80b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80af940 con 0x7fb0e814be90 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80b0a70 con 0x7fb0e80b1730 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80b0d50 con 0x7fb0e80b4d60 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:36.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.716+0000 7fb0ef90e640 10 monclient(hunting): authenticate will time out at 8152.011245s 2024-03-25T18:20:36.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.723+0000 7fb0ecea9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb0e80b1730 0x7fb0e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:45672/0 (socket says 172.21.15.87:45672) 2024-03-25T18:20:36.724 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.723+0000 7fb0ecea9640 1 -- 172.21.15.87:0/1625105400 learned_addr learned my addr 172.21.15.87:0/1625105400 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118509807 0 0) 0x7fb0e80b0a70 con 0x7fb0e80b1730 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:20:36.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.734+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d4003520 con 0x7fb0e80b1730 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540540971 0 0) 0x7fb0e80b0d50 con 0x7fb0e80b4d60 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:20:36.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.743+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80b0a70 con 0x7fb0e80b4d60 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206960391 0 0) 0x7fb0e80af940 con 0x7fb0e814be90 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:20:36.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.747+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80b0d50 con 0x7fb0e814be90 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8177806 0 0) 0x7fb0d4003520 con 0x7fb0e80b1730 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 >> v1:172.21.15.87:6790/0 conn(0x7fb0e80b4d60 legacy=0x7fb0e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 >> v1:172.21.15.87:6789/0 conn(0x7fb0e814be90 legacy=0x7fb0e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 monclient: found mon.noname-c 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80b1030 con 0x7fb0e80b1730 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: _finish_auth 0 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:36.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb0d8001690 con 0x7fb0e80b1730 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0dffff640 10 monclient: dump: 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0ef90e640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:20:36.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.750+0000 7fb0ef90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:36.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d80017f0 con 0x7fb0e80b1730 2024-03-25T18:20:36.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:36.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 1 -- 172.21.15.87:0/1625105400 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb0d8001c00 con 0x7fb0e80b1730 2024-03-25T18:20:36.767 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0ef90e640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0ef90e640 4 set_mon_vals no callback set 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.766+0000 7fb0dffff640 10 monclient: dump: 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:36.768 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.767+0000 7fb0ef90e640 10 monclient: shutdown 2024-03-25T18:20:36.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.767+0000 7fb0ef90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:36.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.767+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1625105400 >> v1:172.21.15.171:6789/0 conn(0x7fb0e80b1730 legacy=0x7fb0e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:36.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.767+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1625105400 shutdown_connections 2024-03-25T18:20:36.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.767+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1625105400 >> 172.21.15.87:0/1625105400 conn(0x7fb0e80617a0 msgr2=0x7fb0e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.768+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1625105400 shutdown_connections 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.768+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1625105400 wait complete. 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.768+0000 7fb0ef90e640 10 monclient: build_initial_monmap 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.768+0000 7fb0ef90e640 10 monclient: monmap: 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:36.770 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:36.771 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 librados: starting msgr at 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 librados: starting objecter 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 Processor -- start 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 -- start start 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 librados: setting wanted keys 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 1 librados: calling monclient init 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.769+0000 7fb0ef90e640 10 monclient: init 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e80b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e80b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e8137500 con 0x7fb0e814be90 2024-03-25T18:20:36.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e81d7940 con 0x7fb0e80b4d60 2024-03-25T18:20:36.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e81d8570 con 0x7fb0e80b1730 2024-03-25T18:20:36.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:36.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.770+0000 7fb0ef90e640 10 monclient(hunting): authenticate will time out at 8152.065024s 2024-03-25T18:20:36.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.775+0000 7fb0edeab640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fb0e814be90 0x7fb0e81380a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:49226/0 (socket says 172.21.15.87:49226) 2024-03-25T18:20:36.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.775+0000 7fb0edeab640 1 -- 172.21.15.87:0/1314867806 learned_addr learned my addr 172.21.15.87:0/1314867806 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:36.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.776+0000 7fb0ddffb640 10 client.?.objecter ms_handle_connect 0x7fb0e814be90 2024-03-25T18:20:36.777 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.776+0000 7fb0ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:36.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.781+0000 7fb0ddffb640 10 client.?.objecter ms_handle_connect 0x7fb0e80b1730 2024-03-25T18:20:36.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.781+0000 7fb0ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:36.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.783+0000 7fb0ddffb640 10 client.?.objecter ms_handle_connect 0x7fb0e80b4d60 2024-03-25T18:20:36.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.783+0000 7fb0ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639887449 0 0) 0x7fb0e8137500 con 0x7fb0e814be90 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:20:36.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.787+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc004eb0 con 0x7fb0e814be90 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302595876 0 0) 0x7fb0e81d7940 con 0x7fb0e80b4d60 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:20:36.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.793+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e8137500 con 0x7fb0e80b4d60 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147698521 0 0) 0x7fb0e81d8570 con 0x7fb0e80b1730 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:20:36.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.798+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e81d7940 con 0x7fb0e80b1730 2024-03-25T18:20:36.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.801+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551511030 0 0) 0x7fb0e8137500 con 0x7fb0e80b4d60 2024-03-25T18:20:36.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.801+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e81d8570 con 0x7fb0e80b4d60 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145620452 0 0) 0x7fb0cc004eb0 con 0x7fb0e814be90 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e8137500 con 0x7fb0e814be90 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb0e0001c70 con 0x7fb0e814be90 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): dump: 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:36.804 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb0d8001b00 con 0x7fb0e80b4d60 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.803+0000 7fb0ddffb640 10 monclient(hunting): dump: 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:36.805 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:36.806 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:36.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767242486 0 0) 0x7fb0e8137500 con 0x7fb0e814be90 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 >> v1:172.21.15.87:6790/0 conn(0x7fb0e80b1730 legacy=0x7fb0e8135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 >> v1:172.21.15.171:6789/0 conn(0x7fb0e80b4d60 legacy=0x7fb0e81d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 1 monclient: found mon.a 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e81d8700 con 0x7fb0e814be90 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 10 monclient: _renew_subs 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:36.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e81387b0 con 0x7fb0e814be90 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 10 monclient: _renew_subs 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8137b00 con 0x7fb0e814be90 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.813+0000 7fb0ef90e640 1 librados: init done 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.814+0000 7fb0ef90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:36.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.814+0000 7fb0ef90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:36.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.814+0000 7fb0ef90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:36.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:36.814+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8141ac0 con 0x7fb0e814be90 2024-03-25T18:20:37.120 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e0003970 con 0x7fb0e814be90 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb0e0003be0 con 0x7fb0e814be90 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0ddffb640 10 monclient: dump: 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:37.121 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.119+0000 7fb0df7fe640 4 set_mon_vals no callback set 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.120+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fb0e0003e00 con 0x7fb0e814be90 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.121+0000 7fb0ed6aa640 10 monclient: get_auth_request con 0x7fb0cc06dfc0 auth_method 0 2024-03-25T18:20:37.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.121+0000 7fb0ddffb640 10 client.4491.objecter ms_handle_connect 0x7fb0cc06dfc0 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fb0e00548a0 con 0x7fb0e814be90 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 10 client.4491.objecter ms_dispatch 0x7fb0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0e00503b0 con 0x7fb0e814be90 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:37.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.126+0000 7fb0ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:37.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.375+0000 7fb0ef90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:37.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.375+0000 7fb0ef90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:37.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.375+0000 7fb0ef90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:20:37.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.375+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e80627b0 con 0x7fb0e814be90 2024-03-25T18:20:37.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.404+0000 7fb0ddffb640 1 -- 172.21.15.87:0/1314867806 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fb0e00522e0 con 0x7fb0e814be90 2024-03-25T18:20:37.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.404+0000 7fb0ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:37.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.404+0000 7fb0ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:37.406 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:37.406 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.406+0000 7fb0ef90e640 10 librados: watch_flush enter 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 10 librados: watch_flush exit 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 >> v1:172.21.15.171:6800/3417775508 conn(0x7fb0cc06dfc0 legacy=0x7fb0cc070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 10 monclient: shutdown 2024-03-25T18:20:37.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 >> v1:172.21.15.87:6789/0 conn(0x7fb0e814be90 legacy=0x7fb0e81380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 shutdown_connections 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 >> 172.21.15.87:0/1314867806 conn(0x7fb0e80617a0 msgr2=0x7fb0e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 shutdown_connections 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 -- 172.21.15.87:0/1314867806 wait complete. 2024-03-25T18:20:37.409 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:37.407+0000 7fb0ef90e640 1 librados: shutdown 2024-03-25T18:20:37.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:44.226 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:44.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.338+0000 7f2d28e86640 10 monclient: get_monmap_and_config 2024-03-25T18:20:44.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.338+0000 7f2d28e86640 10 monclient: build_initial_monmap 2024-03-25T18:20:44.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.338+0000 7f2d28e86640 10 monclient: monmap: 2024-03-25T18:20:44.339 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:44.339 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:44.339253+0000 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:44.339253+0000 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:44.341 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 1 Processor -- start 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 1 -- start start 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 10 monclient: init 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-c con 0x7f2d240a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.339+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-b con 0x7f2d241430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:44.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d24142520 con 0x7f2d2413f710 2024-03-25T18:20:44.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240b28a0 con 0x7f2d240a98f0 2024-03-25T18:20:44.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240b2a30 con 0x7f2d241430c0 2024-03-25T18:20:44.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:44.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.340+0000 7f2d28e86640 10 monclient(hunting): authenticate will time out at 8159.634887s 2024-03-25T18:20:44.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.343+0000 7f2d2259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2d240a98f0 0x7f2d240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:59958/0 (socket says 172.21.15.87:59958) 2024-03-25T18:20:44.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.343+0000 7f2d2259c640 1 -- 172.21.15.87:0/307323338 learned_addr learned my addr 172.21.15.87:0/307323338 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:44.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202151798 0 0) 0x7f2d24142520 con 0x7f2d2413f710 2024-03-25T18:20:44.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.356+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c003520 con 0x7f2d2413f710 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702099894 0 0) 0x7f2d240b2a30 con 0x7f2d241430c0 2024-03-25T18:20:44.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.357+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d24142520 con 0x7f2d241430c0 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496754568 0 0) 0x7f2d240b28a0 con 0x7f2d240a98f0 2024-03-25T18:20:44.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:20:44.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.358+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d240b2a30 con 0x7f2d240a98f0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876454027 0 0) 0x7f2d24142520 con 0x7f2d241430c0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 >> v1:172.21.15.171:6789/0 conn(0x7f2d240a98f0 legacy=0x7f2d240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 >> v1:172.21.15.87:6789/0 conn(0x7f2d2413f710 legacy=0x7f2d241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 monclient: found mon.noname-b 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240b44c0 con 0x7f2d241430c0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: _finish_auth 0 2024-03-25T18:20:44.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d1c0019f0 con 0x7f2d241430c0 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d28e86640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d28e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d2159a640 10 monclient: dump: 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.370 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.371 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.368+0000 7f2d28e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d1c001b50 con 0x7f2d241430c0 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 1 -- 172.21.15.87:0/307323338 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d1c001f60 con 0x7f2d241430c0 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d28e86640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d28e86640 4 set_mon_vals no callback set 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d2159a640 10 monclient: dump: 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.378 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.379 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d28e86640 10 monclient: shutdown 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d28e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.377+0000 7f2d28e86640 1 -- 172.21.15.87:0/307323338 >> v1:172.21.15.87:6790/0 conn(0x7f2d241430c0 legacy=0x7f2d2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.378+0000 7f2d28e86640 1 -- 172.21.15.87:0/307323338 shutdown_connections 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.378+0000 7f2d28e86640 1 -- 172.21.15.87:0/307323338 >> 172.21.15.87:0/307323338 conn(0x7f2d240b7ea0 msgr2=0x7f2d240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.378+0000 7f2d28e86640 1 -- 172.21.15.87:0/307323338 shutdown_connections 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.378+0000 7f2d28e86640 1 -- 172.21.15.87:0/307323338 wait complete. 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 10 monclient: build_initial_monmap 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 10 monclient: monmap: 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:44.381 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 librados: starting msgr at 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 librados: starting objecter 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 Processor -- start 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 -- start start 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 librados: setting wanted keys 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 1 librados: calling monclient init 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.379+0000 7f2d28e86640 10 monclient: init 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-c con 0x7f2d240a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:44.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient(hunting): picked mon.noname-a con 0x7f2d241430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d24137440 con 0x7f2d241430c0 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d24137690 con 0x7f2d2413f710 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d241d7ca0 con 0x7f2d240a98f0 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:44.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.380+0000 7f2d28e86640 10 monclient(hunting): authenticate will time out at 8159.674950s 2024-03-25T18:20:44.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.388+0000 7f2d2259c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f2d240a98f0 0x7f2d24139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39924/0 (socket says 172.21.15.87:39924) 2024-03-25T18:20:44.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.388+0000 7f2d2259c640 1 -- 172.21.15.87:0/2682354377 learned_addr learned my addr 172.21.15.87:0/2682354377 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:44.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.389+0000 7f2d12ffd640 10 client.?.objecter ms_handle_connect 0x7f2d240a98f0 2024-03-25T18:20:44.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.389+0000 7f2d12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:44.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.392+0000 7f2d12ffd640 10 client.?.objecter ms_handle_connect 0x7f2d2413f710 2024-03-25T18:20:44.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.392+0000 7f2d12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190486947 0 0) 0x7f2d241d7ca0 con 0x7f2d240a98f0 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:20:44.399 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.398+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d04004eb0 con 0x7f2d240a98f0 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335952859 0 0) 0x7f2d24137690 con 0x7f2d2413f710 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:20:44.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.399+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d241d7ca0 con 0x7f2d2413f710 2024-03-25T18:20:44.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.401+0000 7f2d12ffd640 10 client.?.objecter ms_handle_connect 0x7f2d241430c0 2024-03-25T18:20:44.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.401+0000 7f2d12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:44.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.405+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885109208 0 0) 0x7f2d24137440 con 0x7f2d241430c0 2024-03-25T18:20:44.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.406+0000 7f2d12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:44.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.406+0000 7f2d12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:44.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.406+0000 7f2d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:44.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.406+0000 7f2d12ffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:20:44.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.406+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d24137690 con 0x7f2d241430c0 2024-03-25T18:20:44.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359182547 0 0) 0x7f2d241d7ca0 con 0x7f2d2413f710 2024-03-25T18:20:44.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d24137440 con 0x7f2d2413f710 2024-03-25T18:20:44.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d18001950 con 0x7f2d2413f710 2024-03-25T18:20:44.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.414+0000 7f2d12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2377602637 0 0) 0x7f2d04004eb0 con 0x7f2d240a98f0 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d241d7ca0 con 0x7f2d240a98f0 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d14001910 con 0x7f2d240a98f0 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:44.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.421+0000 7f2d12ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.423 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057661094 0 0) 0x7f2d24137690 con 0x7f2d241430c0 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d04004eb0 con 0x7f2d241430c0 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d1c001e60 con 0x7f2d241430c0 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.433+0000 7f2d12ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:44.434 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.435 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 315134048 0 0) 0x7f2d24137440 con 0x7f2d2413f710 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 >> v1:172.21.15.87:6790/0 conn(0x7f2d240a98f0 legacy=0x7f2d24139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 >> v1:172.21.15.87:6789/0 conn(0x7f2d241430c0 legacy=0x7f2d24132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 1 monclient: found mon.b 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d24136b60 con 0x7f2d2413f710 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:44.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 10 monclient: _renew_subs 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d24137bb0 con 0x7f2d2413f710 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 10 monclient: _renew_subs 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d240b7330 con 0x7f2d2413f710 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.435+0000 7f2d28e86640 1 librados: init done 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.436+0000 7f2d28e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:44.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.436+0000 7f2d28e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:44.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.436+0000 7f2d28e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:44.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.436+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24137820 con 0x7f2d2413f710 2024-03-25T18:20:44.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d18001d60 con 0x7f2d2413f710 2024-03-25T18:20:44.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:44.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2d180037b0 con 0x7f2d2413f710 2024-03-25T18:20:44.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:44.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d12ffd640 10 monclient: dump: 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.459+0000 7f2d20d99640 4 set_mon_vals no callback set 2024-03-25T18:20:44.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.460+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2d18019800 con 0x7f2d2413f710 2024-03-25T18:20:44.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.461+0000 7f2d2259c640 10 monclient: get_auth_request con 0x7f2d0406e180 auth_method 0 2024-03-25T18:20:44.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.461+0000 7f2d12ffd640 10 client.4498.objecter ms_handle_connect 0x7f2d0406e180 2024-03-25T18:20:44.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.463+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f2d18053520 con 0x7f2d2413f710 2024-03-25T18:20:44.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.463+0000 7f2d12ffd640 10 client.4498.objecter ms_dispatch 0x7f2d24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:44.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.463+0000 7f2d12ffd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:44.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.463+0000 7f2d12ffd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:44.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.463+0000 7f2d12ffd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:20:44.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.737+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d1804f030 con 0x7f2d2413f710 2024-03-25T18:20:44.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.737+0000 7f2d12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:44.738 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.737+0000 7f2d12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:44.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.967+0000 7f2d28e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:44.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.967+0000 7f2d28e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:44.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.967+0000 7f2d28e86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:44.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.967+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d24137d40 con 0x7f2d2413f710 2024-03-25T18:20:44.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.996+0000 7f2d12ffd640 1 -- 172.21.15.87:0/2682354377 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f2d18050f60 con 0x7f2d2413f710 2024-03-25T18:20:44.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.996+0000 7f2d12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:44.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.996+0000 7f2d12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:44.998 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:44.998 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 10 librados: watch_flush enter 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 10 librados: watch_flush exit 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 >> v1:172.21.15.171:6800/3417775508 conn(0x7f2d0406e180 legacy=0x7f2d04070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 10 monclient: shutdown 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 >> v1:172.21.15.171:6789/0 conn(0x7f2d2413f710 legacy=0x7f2d241d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 shutdown_connections 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.998+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 >> 172.21.15.87:0/2682354377 conn(0x7f2d240b7ea0 msgr2=0x7f2d24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:45.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.999+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 shutdown_connections 2024-03-25T18:20:45.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.999+0000 7f2d28e86640 1 -- 172.21.15.87:0/2682354377 wait complete. 2024-03-25T18:20:45.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:44.999+0000 7f2d28e86640 1 librados: shutdown 2024-03-25T18:20:45.012 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:47.428 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:35.141 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.476 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:35.189 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.519 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:35.233 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.550 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:35.229 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.555 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:35.233 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.560 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:35.273 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.566 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:35.279 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.585 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:35.263 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.602 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:35.315 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:51.819 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:51.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.933+0000 7f04f9dd6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:51.934 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.933+0000 7f04f9dd6640 10 monclient: build_initial_monmap 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.933+0000 7f04f9dd6640 10 monclient: monmap: 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:51.934139+0000 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:51.934139+0000 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:51.935 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:51.936 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:51.936 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:51.936 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:51.936 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 1 Processor -- start 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 1 -- start start 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient: init 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:51.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f04f40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f04f413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f04f41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f4142520 con 0x7f04f41430c0 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f40b28a0 con 0x7f04f413f710 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f40b2a30 con 0x7f04f40a98f0 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:51.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.934+0000 7f04f9dd6640 10 monclient(hunting): authenticate will time out at 8167.229778s 2024-03-25T18:20:51.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.939+0000 7f04f37fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f04f40a98f0 0x7f04f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39226/0 (socket says 172.21.15.87:39226) 2024-03-25T18:20:51.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.939+0000 7f04f37fe640 1 -- 172.21.15.87:0/1734690985 learned_addr learned my addr 172.21.15.87:0/1734690985 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:51.955 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727811377 0 0) 0x7f04f40b2a30 con 0x7f04f40a98f0 2024-03-25T18:20:51.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:51.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:51.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:51.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:20:51.956 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.954+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc003520 con 0x7f04f40a98f0 2024-03-25T18:20:51.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454606541 0 0) 0x7f04f40b28a0 con 0x7f04f413f710 2024-03-25T18:20:51.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:51.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:51.957 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:51.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:20:51.958 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.956+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f40b2a30 con 0x7f04f413f710 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.965+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494172133 0 0) 0x7f04f4142520 con 0x7f04f41430c0 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.966+0000 7f04f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.966+0000 7f04f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.966+0000 7f04f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.966+0000 7f04f27fc640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:20:51.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.966+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f40b28a0 con 0x7f04f41430c0 2024-03-25T18:20:51.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169443330 0 0) 0x7f04f40b2a30 con 0x7f04f413f710 2024-03-25T18:20:51.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 >> v1:172.21.15.87:6790/0 conn(0x7f04f40a98f0 legacy=0x7f04f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 >> v1:172.21.15.87:6789/0 conn(0x7f04f41430c0 legacy=0x7f04f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 1 monclient: found mon.noname-c 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40b44c0 con 0x7f04f413f710 2024-03-25T18:20:51.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f9dd6640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.968+0000 7f04f9dd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.969+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f04e8001630 con 0x7f04f413f710 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.969+0000 7f04f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:51.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.969+0000 7f04f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.969+0000 7f04f27fc640 10 monclient: dump: 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:51.973 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:51.974 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:51.974 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:51.974 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:51.974 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:51.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.969+0000 7f04f9dd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:51.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e8001880 con 0x7f04f413f710 2024-03-25T18:20:51.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:51.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 1 -- 172.21.15.87:0/1734690985 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f04e8001c10 con 0x7f04f413f710 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f9dd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f27fc640 10 monclient: dump: 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.972+0000 7f04f9dd6640 4 set_mon_vals no callback set 2024-03-25T18:20:51.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 10 monclient: shutdown 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 1 -- 172.21.15.87:0/1734690985 >> v1:172.21.15.171:6789/0 conn(0x7f04f413f710 legacy=0x7f04f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 1 -- 172.21.15.87:0/1734690985 shutdown_connections 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 1 -- 172.21.15.87:0/1734690985 >> 172.21.15.87:0/1734690985 conn(0x7f04f40b7ea0 msgr2=0x7f04f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 1 -- 172.21.15.87:0/1734690985 shutdown_connections 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 1 -- 172.21.15.87:0/1734690985 wait complete. 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 10 monclient: build_initial_monmap 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.973+0000 7f04f9dd6640 10 monclient: monmap: 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:51.980 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 librados: starting msgr at 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 librados: starting objecter 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 Processor -- start 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 -- start start 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 librados: setting wanted keys 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 1 librados: calling monclient init 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.974+0000 7f04f9dd6640 10 monclient: init 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f04f40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f04f413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f04f41430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:51.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f4136c60 con 0x7f04f413f710 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f4135970 con 0x7f04f41430c0 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f4136ed0 con 0x7f04f40a98f0 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.975+0000 7f04f9dd6640 10 monclient(hunting): authenticate will time out at 8167.270526s 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.981+0000 7f04f37fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f04f40a98f0 0x7f04f4133330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:39250/0 (socket says 172.21.15.87:39250) 2024-03-25T18:20:51.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.981+0000 7f04f37fe640 1 -- 172.21.15.87:0/2402033275 learned_addr learned my addr 172.21.15.87:0/2402033275 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:51.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.986+0000 7f04dbfff640 10 client.?.objecter ms_handle_connect 0x7f04f40a98f0 2024-03-25T18:20:51.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.986+0000 7f04dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:52.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.999+0000 7f04dbfff640 10 client.?.objecter ms_handle_connect 0x7f04f41430c0 2024-03-25T18:20:52.000 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:51.999+0000 7f04dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:52.004 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.003+0000 7f04dbfff640 10 client.?.objecter ms_handle_connect 0x7f04f413f710 2024-03-25T18:20:52.004 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.003+0000 7f04dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:52.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237783502 0 0) 0x7f04f4136ed0 con 0x7f04f40a98f0 2024-03-25T18:20:52.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:52.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:52.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:52.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:20:52.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.008+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e0004eb0 con 0x7f04f40a98f0 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544757782 0 0) 0x7f04f4136c60 con 0x7f04f413f710 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:20:52.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.022+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4136ed0 con 0x7f04f413f710 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628861919 0 0) 0x7f04f4135970 con 0x7f04f41430c0 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:20:52.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.025+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4136c60 con 0x7f04f41430c0 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247618121 0 0) 0x7f04e0004eb0 con 0x7f04f40a98f0 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f4135970 con 0x7f04f40a98f0 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f04e4001950 con 0x7f04f40a98f0 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 10 monclient(hunting): dump: 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:52.033 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:52.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.032+0000 7f04dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732038478 0 0) 0x7f04f4136c60 con 0x7f04f41430c0 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e0004eb0 con 0x7f04f41430c0 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f04ec001bd0 con 0x7f04f41430c0 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.034+0000 7f04dbfff640 10 monclient(hunting): dump: 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:52.035 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:52.036 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:52.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 736144175 0 0) 0x7f04e0004eb0 con 0x7f04f41430c0 2024-03-25T18:20:52.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 >> v1:172.21.15.87:6790/0 conn(0x7f04f40a98f0 legacy=0x7f04f4133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:52.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 >> v1:172.21.15.87:6789/0 conn(0x7f04f413f710 legacy=0x7f04f41378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 1 monclient: found mon.b 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f41371b0 con 0x7f04f41430c0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 10 monclient: _renew_subs 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f4137490 con 0x7f04f41430c0 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:52.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 10 monclient: _renew_subs 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f4135270 con 0x7f04f41430c0 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.047+0000 7f04f9dd6640 1 librados: init done 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.048+0000 7f04f9dd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.048+0000 7f04f9dd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:52.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.048+0000 7f04f9dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:52.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.048+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f40627b0 con 0x7f04f41430c0 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04ec001fe0 con 0x7f04f41430c0 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f04ec003b30 con 0x7f04f41430c0 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04dbfff640 10 monclient: dump: 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:52.334 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.331+0000 7f04f1ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:52.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.332+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f04ec019b00 con 0x7f04f41430c0 2024-03-25T18:20:52.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.333+0000 7f04f37fe640 10 monclient: get_auth_request con 0x7f04e006dfd0 auth_method 0 2024-03-25T18:20:52.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.333+0000 7f04dbfff640 10 client.4504.objecter ms_handle_connect 0x7f04e006dfd0 2024-03-25T18:20:52.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f04ec053770 con 0x7f04f41430c0 2024-03-25T18:20:52.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 10 client.4504.objecter ms_dispatch 0x7f04f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:52.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:52.339 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:52.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:20:52.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04ec04f280 con 0x7f04f41430c0 2024-03-25T18:20:52.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:52.340 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.337+0000 7f04dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:52.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.592+0000 7f04f9dd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:52.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.592+0000 7f04f9dd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:52.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.592+0000 7f04f9dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:20:52.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.592+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f413b430 con 0x7f04f41430c0 2024-03-25T18:20:52.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.645+0000 7f04dbfff640 1 -- 172.21.15.87:0/2402033275 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f04ec0511b0 con 0x7f04f41430c0 2024-03-25T18:20:52.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.645+0000 7f04dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:52.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.645+0000 7f04dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:52.648 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:20:52.648 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:52.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.647+0000 7f04f9dd6640 10 librados: watch_flush enter 2024-03-25T18:20:52.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 10 librados: watch_flush exit 2024-03-25T18:20:52.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:20:52.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:20:52.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 >> v1:172.21.15.171:6800/3417775508 conn(0x7f04e006dfd0 legacy=0x7f04e0070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 10 monclient: shutdown 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 >> v1:172.21.15.171:6789/0 conn(0x7f04f41430c0 legacy=0x7f04f41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 shutdown_connections 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 >> 172.21.15.87:0/2402033275 conn(0x7f04f40b7ea0 msgr2=0x7f04f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 shutdown_connections 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 -- 172.21.15.87:0/2402033275 wait complete. 2024-03-25T18:20:52.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:52.648+0000 7f04f9dd6640 1 librados: shutdown 2024-03-25T18:20:52.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:59.469 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:59.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.578+0000 7f512cae2640 10 monclient: get_monmap_and_config 2024-03-25T18:20:59.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.578+0000 7f512cae2640 10 monclient: build_initial_monmap 2024-03-25T18:20:59.580 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.578+0000 7f512cae2640 10 monclient: monmap: 2024-03-25T18:20:59.580 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:59.580 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:20:59.580520+0000 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:20:59.580520+0000 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:20:59.581 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.582 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 1 Processor -- start 2024-03-25T18:20:59.582 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 1 -- start start 2024-03-25T18:20:59.582 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 10 monclient: init 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-b con 0x7f51280b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.579+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-c con 0x7f51280b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-a con 0x7f512814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51280af940 con 0x7f512814be90 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51280b0a70 con 0x7f51280b1730 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51280b0d50 con 0x7f51280b4d60 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:59.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.580+0000 7f512cae2640 10 monclient(hunting): authenticate will time out at 8174.876018s 2024-03-25T18:20:59.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.589+0000 7f511dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f51280b1730 0x7f51280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:34526/0 (socket says 172.21.15.87:34526) 2024-03-25T18:20:59.591 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.589+0000 7f511dd9b640 1 -- 172.21.15.87:0/3589833643 learned_addr learned my addr 172.21.15.87:0/3589833643 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:59.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661283537 0 0) 0x7f51280b0a70 con 0x7f51280b1730 2024-03-25T18:20:59.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:59.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:59.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:59.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:20:59.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.601+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5114003520 con 0x7f51280b1730 2024-03-25T18:20:59.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516547886 0 0) 0x7f51280b0d50 con 0x7f51280b4d60 2024-03-25T18:20:59.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:59.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:59.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:59.625 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:20:59.625 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.622+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51280b0a70 con 0x7f51280b4d60 2024-03-25T18:20:59.628 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397958772 0 0) 0x7f5114003520 con 0x7f51280b1730 2024-03-25T18:20:59.628 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 >> v1:172.21.15.87:6790/0 conn(0x7f51280b4d60 legacy=0x7f512813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:59.628 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 >> v1:172.21.15.87:6789/0 conn(0x7f512814be90 legacy=0x7f512814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:59.628 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 monclient: found mon.noname-c 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51280b1030 con 0x7f51280b1730 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5110001690 con 0x7f51280b1730 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f5125d9b640 10 monclient: dump: 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.629 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f512cae2640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:20:59.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.626+0000 7f512cae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:59.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51100017f0 con 0x7f51280b1730 2024-03-25T18:20:59.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:59.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 1 -- 172.21.15.87:0/3589833643 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5110001c00 con 0x7f51280b1730 2024-03-25T18:20:59.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f512cae2640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f512cae2640 4 set_mon_vals no callback set 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.646+0000 7f5125d9b640 10 monclient: dump: 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:59.649 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 10 monclient: shutdown 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 1 -- 172.21.15.87:0/3589833643 >> v1:172.21.15.171:6789/0 conn(0x7f51280b1730 legacy=0x7f51280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 1 -- 172.21.15.87:0/3589833643 shutdown_connections 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 1 -- 172.21.15.87:0/3589833643 >> 172.21.15.87:0/3589833643 conn(0x7f51280617a0 msgr2=0x7f5128061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:59.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 1 -- 172.21.15.87:0/3589833643 shutdown_connections 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.647+0000 7f512cae2640 1 -- 172.21.15.87:0/3589833643 wait complete. 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.648+0000 7f512cae2640 10 monclient: build_initial_monmap 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.648+0000 7f512cae2640 10 monclient: monmap: 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 librados: starting msgr at 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 librados: starting objecter 2024-03-25T18:20:59.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 Processor -- start 2024-03-25T18:20:59.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 -- start start 2024-03-25T18:20:59.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 librados: setting wanted keys 2024-03-25T18:20:59.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 1 librados: calling monclient init 2024-03-25T18:20:59.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.649+0000 7f512cae2640 10 monclient: init 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-b con 0x7f51280b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-c con 0x7f51280b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient(hunting): picked mon.noname-a con 0x7f512814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128135810 con 0x7f512814be90 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128144c10 con 0x7f51280b1730 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128144ef0 con 0x7f51280b4d60 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:59.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.650+0000 7f512cae2640 10 monclient(hunting): authenticate will time out at 8174.946411s 2024-03-25T18:20:59.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.661+0000 7f511dd9b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f51280b4d60 0x7f51281349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:51622/0 (socket says 172.21.15.87:51622) 2024-03-25T18:20:59.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.661+0000 7f511dd9b640 1 -- 172.21.15.87:0/287604643 learned_addr learned my addr 172.21.15.87:0/287604643 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:20:59.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.663+0000 7f511f7fe640 10 client.?.objecter ms_handle_connect 0x7f51280b4d60 2024-03-25T18:20:59.665 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.663+0000 7f511f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:59.665 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.663+0000 7f511f7fe640 10 client.?.objecter ms_handle_connect 0x7f51280b1730 2024-03-25T18:20:59.665 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.663+0000 7f511f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217850790 0 0) 0x7f5128144ef0 con 0x7f51280b4d60 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:20:59.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.668+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5108004eb0 con 0x7f51280b4d60 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052757475 0 0) 0x7f5128144c10 con 0x7f51280b1730 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5128144ef0 con 0x7f51280b1730 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 client.?.objecter ms_handle_connect 0x7f512814be90 2024-03-25T18:20:59.678 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.676+0000 7f511f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:59.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009202410 0 0) 0x7f5128135810 con 0x7f512814be90 2024-03-25T18:20:59.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:59.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:59.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:59.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:20:59.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.687+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5128144c10 con 0x7f512814be90 2024-03-25T18:20:59.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017403869 0 0) 0x7f5108004eb0 con 0x7f51280b4d60 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5128135810 con 0x7f51280b4d60 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5110001bc0 con 0x7f51280b4d60 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.688+0000 7f511f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:59.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491548242 0 0) 0x7f5128144ef0 con 0x7f51280b1730 2024-03-25T18:20:59.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5108004eb0 con 0x7f51280b1730 2024-03-25T18:20:59.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f51180019f0 con 0x7f51280b1730 2024-03-25T18:20:59.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.702+0000 7f511f7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 263357087 0 0) 0x7f5128135810 con 0x7f51280b4d60 2024-03-25T18:20:59.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 >> v1:172.21.15.171:6789/0 conn(0x7f51280b1730 legacy=0x7f51281d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 >> v1:172.21.15.87:6789/0 conn(0x7f512814be90 legacy=0x7f5128135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 1 monclient: found mon.c 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128140c10 con 0x7f51280b4d60 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f511f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 10 monclient: _renew_subs 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5128137d50 con 0x7f51280b4d60 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 10 monclient: _renew_subs 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:20:59.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51281407d0 con 0x7f51280b4d60 2024-03-25T18:20:59.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.703+0000 7f512cae2640 1 librados: init done 2024-03-25T18:20:59.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.704+0000 7f512cae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:59.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.704+0000 7f512cae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:59.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.704+0000 7f512cae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:20:59.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.704+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5128141e30 con 0x7f51280b4d60 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51100011f0 con 0x7f51280b4d60 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5110004cb0 con 0x7f51280b4d60 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f511f7fe640 10 monclient: dump: 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:20:59.710 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:20:59.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.708+0000 7f512559a640 4 set_mon_vals no callback set 2024-03-25T18:20:59.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.811+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f511001ac80 con 0x7f51280b4d60 2024-03-25T18:20:59.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.812+0000 7f512659c640 10 monclient: get_auth_request con 0x7f510806ded0 auth_method 0 2024-03-25T18:20:59.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.812+0000 7f511f7fe640 10 client.4514.objecter ms_handle_connect 0x7f510806ded0 2024-03-25T18:20:59.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.816+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f51100549b0 con 0x7f51280b4d60 2024-03-25T18:20:59.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.816+0000 7f511f7fe640 10 client.4514.objecter ms_dispatch 0x7f5128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:59.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.816+0000 7f511f7fe640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:59.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.816+0000 7f511f7fe640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:59.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:20:59.816+0000 7f511f7fe640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T18:21:00.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.006+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51100502f0 con 0x7f51280b4d60 2024-03-25T18:21:00.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.006+0000 7f511f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:00.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.006+0000 7f511f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.241+0000 7f512cae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:00.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.241+0000 7f512cae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.241+0000 7f512cae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:00.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.241+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51281380a0 con 0x7f51280b4d60 2024-03-25T18:21:00.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.318+0000 7f511f7fe640 1 -- 172.21.15.87:0/287604643 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f51281380a0 con 0x7f51280b4d60 2024-03-25T18:21:00.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.318+0000 7f511f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:00.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.318+0000 7f511f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:00.320 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:00.320 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.320+0000 7f512cae2640 10 librados: watch_flush enter 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.320+0000 7f512cae2640 10 librados: watch_flush exit 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.320+0000 7f512cae2640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.320+0000 7f512cae2640 10 client.4514.objecter successfully canceled tick 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 >> v1:172.21.15.171:6800/3417775508 conn(0x7f510806ded0 legacy=0x7f5108070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 10 monclient: shutdown 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:00.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 >> v1:172.21.15.87:6790/0 conn(0x7f51280b4d60 legacy=0x7f51281349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:00.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 shutdown_connections 2024-03-25T18:21:00.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 >> 172.21.15.87:0/287604643 conn(0x7f51280617a0 msgr2=0x7f51280b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:00.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 shutdown_connections 2024-03-25T18:21:00.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 -- 172.21.15.87:0/287604643 wait complete. 2024-03-25T18:21:00.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:00.321+0000 7f512cae2640 1 librados: shutdown 2024-03-25T18:21:00.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:07.139 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.256+0000 7f72ab4f6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.256+0000 7f72ab4f6640 10 monclient: build_initial_monmap 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.256+0000 7f72ab4f6640 10 monclient: monmap: 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:07.257352+0000 2024-03-25T18:21:07.257 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:07.257352+0000 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.257+0000 7f72ab4f6640 1 Processor -- start 2024-03-25T18:21:07.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.257+0000 7f72ab4f6640 1 -- start start 2024-03-25T18:21:07.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.257+0000 7f72ab4f6640 10 monclient: init 2024-03-25T18:21:07.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f72a40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f72a413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f72a41430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a4142520 con 0x7f72a413f710 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40b28a0 con 0x7f72a41430c0 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40b2a30 con 0x7f72a40a98f0 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:07.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.258+0000 7f72ab4f6640 10 monclient(hunting): authenticate will time out at 8182.553603s 2024-03-25T18:21:07.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.270+0000 7f72a9292640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f72a40a98f0 0x7f72a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:56528/0 (socket says 172.21.15.87:56528) 2024-03-25T18:21:07.272 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.270+0000 7f72a9292640 1 -- 172.21.15.87:0/2390902462 learned_addr learned my addr 172.21.15.87:0/2390902462 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:07.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117288343 0 0) 0x7f72a40b2a30 con 0x7f72a40a98f0 2024-03-25T18:21:07.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:21:07.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.286+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7290003520 con 0x7f72a40a98f0 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131224835 0 0) 0x7f72a40b28a0 con 0x7f72a41430c0 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:21:07.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.288+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a40b2a30 con 0x7f72a41430c0 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335876554 0 0) 0x7f72a4142520 con 0x7f72a413f710 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:21:07.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.294+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a40b28a0 con 0x7f72a413f710 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.301+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595331122 0 0) 0x7f72a40b28a0 con 0x7f72a413f710 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 >> v1:172.21.15.87:6790/0 conn(0x7f72a40a98f0 legacy=0x7f72a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 >> v1:172.21.15.171:6789/0 conn(0x7f72a41430c0 legacy=0x7f72a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 1 monclient: found mon.noname-a 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40b44c0 con 0x7f72a413f710 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f72940015b0 con 0x7f72a413f710 2024-03-25T18:21:07.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f72ab4f6640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f72ab4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f729bfff640 10 monclient: dump: 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:07.304 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.305 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.302+0000 7f72ab4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7294001710 con 0x7f72a413f710 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 1 -- 172.21.15.87:0/2390902462 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7294001b20 con 0x7f72a413f710 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f72ab4f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f72ab4f6640 4 set_mon_vals no callback set 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.307+0000 7f729bfff640 10 monclient: dump: 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:07.308 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:07.309 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 10 monclient: shutdown 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 1 -- 172.21.15.87:0/2390902462 >> v1:172.21.15.87:6789/0 conn(0x7f72a413f710 legacy=0x7f72a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 1 -- 172.21.15.87:0/2390902462 shutdown_connections 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 1 -- 172.21.15.87:0/2390902462 >> 172.21.15.87:0/2390902462 conn(0x7f72a40b7ea0 msgr2=0x7f72a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 1 -- 172.21.15.87:0/2390902462 shutdown_connections 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.308+0000 7f72ab4f6640 1 -- 172.21.15.87:0/2390902462 wait complete. 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 10 monclient: build_initial_monmap 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 10 monclient: monmap: 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:07.311 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 librados: starting msgr at 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 librados: starting objecter 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 Processor -- start 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 -- start start 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 librados: setting wanted keys 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 1 librados: calling monclient init 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.309+0000 7f72ab4f6640 10 monclient: init 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f72a40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:07.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f72a413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f72a41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a41d8340 con 0x7f72a41430c0 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a41d7710 con 0x7f72a40a98f0 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a4138e90 con 0x7f72a413f710 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:07.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.310+0000 7f72ab4f6640 10 monclient(hunting): authenticate will time out at 8182.605058s 2024-03-25T18:21:07.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.318+0000 7f72a9292640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72a40a98f0 0x7f72a41d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:39330/0 (socket says 172.21.15.87:39330) 2024-03-25T18:21:07.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.318+0000 7f72a9292640 1 -- 172.21.15.87:0/3393786452 learned_addr learned my addr 172.21.15.87:0/3393786452 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:07.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.320+0000 7f7299ffb640 10 client.?.objecter ms_handle_connect 0x7f72a40a98f0 2024-03-25T18:21:07.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.320+0000 7f7299ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325672822 0 0) 0x7f72a41d7710 con 0x7f72a40a98f0 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7288004eb0 con 0x7f72a40a98f0 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 client.?.objecter ms_handle_connect 0x7f72a41430c0 2024-03-25T18:21:07.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.325+0000 7f7299ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:07.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.328+0000 7f7299ffb640 10 client.?.objecter ms_handle_connect 0x7f72a413f710 2024-03-25T18:21:07.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.328+0000 7f7299ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:07.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149435762 0 0) 0x7f72a41d8340 con 0x7f72a41430c0 2024-03-25T18:21:07.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.344 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:21:07.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.343+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a41d7710 con 0x7f72a41430c0 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274444583 0 0) 0x7f72a4138e90 con 0x7f72a413f710 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:21:07.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.354+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a41d8340 con 0x7f72a413f710 2024-03-25T18:21:07.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2244309456 0 0) 0x7f7288004eb0 con 0x7f72a40a98f0 2024-03-25T18:21:07.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72a4138e90 con 0x7f72a40a98f0 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f72a0001950 con 0x7f72a40a98f0 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.359+0000 7f7299ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420396361 0 0) 0x7f72a4138e90 con 0x7f72a40a98f0 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 >> v1:172.21.15.87:6790/0 conn(0x7f72a413f710 legacy=0x7f72a4138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 >> v1:172.21.15.87:6789/0 conn(0x7f72a41430c0 legacy=0x7f72a4133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 1 monclient: found mon.b 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.372+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4139020 con 0x7f72a40a98f0 2024-03-25T18:21:07.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f7299ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f7299ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f7299ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 10 monclient: _renew_subs 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a4137f30 con 0x7f72a40a98f0 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 10 monclient: _renew_subs 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a41348e0 con 0x7f72a40a98f0 2024-03-25T18:21:07.375 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.373+0000 7f72ab4f6640 1 librados: init done 2024-03-25T18:21:07.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.374+0000 7f72ab4f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:07.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.374+0000 7f72ab4f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:07.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.374+0000 7f72ab4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:07.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.374+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a41d78a0 con 0x7f72a40a98f0 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a00047e0 con 0x7f72a40a98f0 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f72a0004af0 con 0x7f72a40a98f0 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f7299ffb640 10 monclient: dump: 2024-03-25T18:21:07.655 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:07.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.654+0000 7f729b7fe640 4 set_mon_vals no callback set 2024-03-25T18:21:07.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.655+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f72a001aac0 con 0x7f72a40a98f0 2024-03-25T18:21:07.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.656+0000 7f72a8a91640 10 monclient: get_auth_request con 0x7f728806ddd0 auth_method 0 2024-03-25T18:21:07.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.656+0000 7f7299ffb640 10 client.4516.objecter ms_handle_connect 0x7f728806ddd0 2024-03-25T18:21:07.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f72a0054840 con 0x7f72a40a98f0 2024-03-25T18:21:07.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 10 client.4516.objecter ms_dispatch 0x7f72a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72a0050180 con 0x7f72a40a98f0 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:07.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.659+0000 7f7299ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:07.713 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:55.427 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.720 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:55.433 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.760 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:06:55.474 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.777 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:55.456 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.781 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:55.459 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.798 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:06:55.477 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.827 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:55.540 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.837 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:55.550 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.864 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:06:55.577 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:07.871 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.870+0000 7f72ab4f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:07.871 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.870+0000 7f72ab4f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:07.871 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.870+0000 7f72ab4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:07.871 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:07.870+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72a414f2d0 con 0x7f72a40a98f0 2024-03-25T18:21:08.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.040+0000 7f7299ffb640 1 -- 172.21.15.87:0/3393786452 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f72a0052280 con 0x7f72a40a98f0 2024-03-25T18:21:08.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.040+0000 7f7299ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:08.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.040+0000 7f7299ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:08.041 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:08.042 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:08.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 10 librados: watch_flush enter 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 10 librados: watch_flush exit 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 >> v1:172.21.15.171:6800/3417775508 conn(0x7f728806ddd0 legacy=0x7f7288070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 10 monclient: shutdown 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 >> v1:172.21.15.171:6789/0 conn(0x7f72a40a98f0 legacy=0x7f72a41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 shutdown_connections 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 >> 172.21.15.87:0/3393786452 conn(0x7f72a40b7ea0 msgr2=0x7f72a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 shutdown_connections 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 -- 172.21.15.87:0/3393786452 wait complete. 2024-03-25T18:21:08.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:08.042+0000 7f72ab4f6640 1 librados: shutdown 2024-03-25T18:21:08.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:14.862 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.974+0000 7f1683372640 10 monclient: get_monmap_and_config 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.974+0000 7f1683372640 10 monclient: build_initial_monmap 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.974+0000 7f1683372640 10 monclient: monmap: 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:14.975088+0000 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:14.975088+0000 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:14.975 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:14.976 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:14.976 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:14.976 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:14.976 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.974+0000 7f1683372640 1 Processor -- start 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 1 -- start start 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient: init 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:14.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient(hunting): picked mon.noname-b con 0x7f167c0b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient(hunting): picked mon.noname-a con 0x7f167c0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient(hunting): picked mon.noname-c con 0x7f167c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0af940 con 0x7f167c0b1730 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0b0a70 con 0x7f167c14be90 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0b0d50 con 0x7f167c0b4d60 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:14.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.975+0000 7f1683372640 10 monclient(hunting): authenticate will time out at 8190.270592s 2024-03-25T18:21:14.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.989+0000 7f168190f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f167c14be90 0x7f167c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:49900/0 (socket says 172.21.15.87:49900) 2024-03-25T18:21:14.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:14.989+0000 7f168190f640 1 -- 172.21.15.87:0/2993299045 learned_addr learned my addr 172.21.15.87:0/2993299045 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:15.005 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90951392 0 0) 0x7f167c0b0a70 con 0x7f167c14be90 2024-03-25T18:21:15.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:21:15.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.004+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1668003520 con 0x7f167c14be90 2024-03-25T18:21:15.009 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.008+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156472553 0 0) 0x7f167c0b0d50 con 0x7f167c0b4d60 2024-03-25T18:21:15.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.008+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.008+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.008+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.008+0000 7f1673fff640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:21:15.010 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.009+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c0b0a70 con 0x7f167c0b4d60 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822174730 0 0) 0x7f167c0af940 con 0x7f167c0b1730 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:21:15.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.010+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c0b0d50 con 0x7f167c0b1730 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94917064 0 0) 0x7f1668003520 con 0x7f167c14be90 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 >> v1:172.21.15.87:6790/0 conn(0x7f167c0b4d60 legacy=0x7f167c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 >> v1:172.21.15.87:6789/0 conn(0x7f167c0b1730 legacy=0x7f167c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 monclient: found mon.noname-c 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0b1030 con 0x7f167c14be90 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:15.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1674001a10 con 0x7f167c14be90 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1673fff640 10 monclient: dump: 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.020 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.021 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1683372640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:21:15.021 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.017+0000 7f1683372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:15.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1674001b70 con 0x7f167c14be90 2024-03-25T18:21:15.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 1 -- 172.21.15.87:0/2993299045 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1674001f80 con 0x7f167c14be90 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1683372640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1683372640 4 set_mon_vals no callback set 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.040+0000 7f1673fff640 10 monclient: dump: 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.042 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.043 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.043 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 10 monclient: shutdown 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 1 -- 172.21.15.87:0/2993299045 >> v1:172.21.15.171:6789/0 conn(0x7f167c14be90 legacy=0x7f167c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 1 -- 172.21.15.87:0/2993299045 shutdown_connections 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 1 -- 172.21.15.87:0/2993299045 >> 172.21.15.87:0/2993299045 conn(0x7f167c0617a0 msgr2=0x7f167c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 1 -- 172.21.15.87:0/2993299045 shutdown_connections 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.041+0000 7f1683372640 1 -- 172.21.15.87:0/2993299045 wait complete. 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 10 monclient: build_initial_monmap 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 10 monclient: monmap: 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:15.045 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 librados: starting msgr at 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 librados: starting objecter 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 Processor -- start 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 -- start start 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 librados: setting wanted keys 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.042+0000 7f1683372640 1 librados: calling monclient init 2024-03-25T18:21:15.046 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient: init 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient(hunting): picked mon.noname-c con 0x7f167c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient(hunting): picked mon.noname-a con 0x7f167c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient(hunting): picked mon.noname-b con 0x7f167c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c1408a0 con 0x7f167c0b4d60 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c140dc0 con 0x7f167c14be90 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c140f50 con 0x7f167c0b1730 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:15.047 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.043+0000 7f1683372640 10 monclient(hunting): authenticate will time out at 8190.338592s 2024-03-25T18:21:15.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.050+0000 7f168090d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f167c0b4d60 0x7f167c1d7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:40396/0 (socket says 172.21.15.87:40396) 2024-03-25T18:21:15.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.050+0000 7f168090d640 1 -- 172.21.15.87:0/2649189753 learned_addr learned my addr 172.21.15.87:0/2649189753 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:15.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.051+0000 7f1671ffb640 10 client.?.objecter ms_handle_connect 0x7f167c0b4d60 2024-03-25T18:21:15.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.051+0000 7f1671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:15.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.065+0000 7f1671ffb640 10 client.?.objecter ms_handle_connect 0x7f167c0b1730 2024-03-25T18:21:15.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.065+0000 7f1671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729751089 0 0) 0x7f167c1408a0 con 0x7f167c0b4d60 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:21:15.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.066+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1660004eb0 con 0x7f167c0b4d60 2024-03-25T18:21:15.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.068+0000 7f1671ffb640 10 client.?.objecter ms_handle_connect 0x7f167c14be90 2024-03-25T18:21:15.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.068+0000 7f1671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846539003 0 0) 0x7f167c140dc0 con 0x7f167c14be90 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:21:15.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.085+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c1408a0 con 0x7f167c14be90 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754041987 0 0) 0x7f167c140f50 con 0x7f167c0b1730 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:21:15.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.091+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c140dc0 con 0x7f167c0b1730 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2198147522 0 0) 0x7f1660004eb0 con 0x7f167c0b4d60 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f167c140f50 con 0x7f167c0b4d60 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f166c001a10 con 0x7f167c0b4d60 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.101 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.102 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.102 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.102 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.102 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.100+0000 7f1671ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583483255 0 0) 0x7f167c1408a0 con 0x7f167c14be90 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1660004eb0 con 0x7f167c14be90 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1674001e80 con 0x7f167c14be90 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.115+0000 7f1671ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.116 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.117 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.117 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.117 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.117 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021042182 0 0) 0x7f167c140dc0 con 0x7f167c0b1730 2024-03-25T18:21:15.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f167c1408a0 con 0x7f167c0b1730 2024-03-25T18:21:15.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1678001930 con 0x7f167c0b1730 2024-03-25T18:21:15.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.124+0000 7f1671ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.126 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4227978425 0 0) 0x7f167c140f50 con 0x7f167c0b4d60 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 >> v1:172.21.15.87:6790/0 conn(0x7f167c0b1730 legacy=0x7f167c134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 >> v1:172.21.15.171:6789/0 conn(0x7f167c14be90 legacy=0x7f167c1e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 1 monclient: found mon.a 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c140380 con 0x7f167c0b4d60 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 10 monclient: _renew_subs 2024-03-25T18:21:15.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c140630 con 0x7f167c0b4d60 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 10 monclient: _renew_subs 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c135730 con 0x7f167c0b4d60 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.127+0000 7f1683372640 1 librados: init done 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.128+0000 7f1683372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:15.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.128+0000 7f1683372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:15.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.128+0000 7f1683372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:21:15.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.128+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c141df0 con 0x7f167c0b4d60 2024-03-25T18:21:15.463 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f166c001c70 con 0x7f167c0b4d60 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f166c003a60 con 0x7f167c0b4d60 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f1671ffb640 10 monclient: dump: 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:15.464 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.462+0000 7f16737fe640 4 set_mon_vals no callback set 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.463+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f166c003c80 con 0x7f167c0b4d60 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.464+0000 7f168110e640 10 monclient: get_auth_request con 0x7f166006e0f0 auth_method 0 2024-03-25T18:21:15.465 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.464+0000 7f1671ffb640 10 client.4521.objecter ms_handle_connect 0x7f166006e0f0 2024-03-25T18:21:15.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f166c054840 con 0x7f167c0b4d60 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 10 client.4521.objecter ms_dispatch 0x7f167c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f166c050350 con 0x7f167c0b4d60 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:15.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.468+0000 7f1671ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:15.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.681+0000 7f1683372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:15.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.681+0000 7f1683372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:15.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.681+0000 7f1683372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:21:15.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.681+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f167c0627b0 con 0x7f167c0b4d60 2024-03-25T18:21:15.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.744+0000 7f1671ffb640 1 -- 172.21.15.87:0/2649189753 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f166c052280 con 0x7f167c0b4d60 2024-03-25T18:21:15.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.744+0000 7f1671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:15.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.744+0000 7f1671ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:15.745 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:15.745 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 10 librados: watch_flush enter 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 10 librados: watch_flush exit 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 >> v1:172.21.15.171:6800/3417775508 conn(0x7f166006e0f0 legacy=0x7f16600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 10 monclient: shutdown 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:15.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 >> v1:172.21.15.87:6789/0 conn(0x7f167c0b4d60 legacy=0x7f167c1d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:15.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 shutdown_connections 2024-03-25T18:21:15.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 >> 172.21.15.87:0/2649189753 conn(0x7f167c0617a0 msgr2=0x7f167c13c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:15.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 shutdown_connections 2024-03-25T18:21:15.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 -- 172.21.15.87:0/2649189753 wait complete. 2024-03-25T18:21:15.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:15.746+0000 7f1683372640 1 librados: shutdown 2024-03-25T18:21:15.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:22.566 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.682+0000 7f8e9e48f640 10 monclient: get_monmap_and_config 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.682+0000 7f8e9e48f640 10 monclient: build_initial_monmap 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.682+0000 7f8e9e48f640 10 monclient: monmap: 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:22.683147+0000 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:22.683147+0000 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:22.683 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:22.684 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:22.684 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.683+0000 7f8e9e48f640 1 Processor -- start 2024-03-25T18:21:22.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.683+0000 7f8e9e48f640 1 -- start start 2024-03-25T18:21:22.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.683+0000 7f8e9e48f640 10 monclient: init 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-c con 0x7f8e980a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-a con 0x7f8e981430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e98142520 con 0x7f8e981430c0 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980b28a0 con 0x7f8e980a98f0 2024-03-25T18:21:22.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980b2a30 con 0x7f8e9813f710 2024-03-25T18:21:22.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:22.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.684+0000 7f8e9e48f640 10 monclient(hunting): authenticate will time out at 8197.979205s 2024-03-25T18:21:22.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.694+0000 7f8e977fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f8e9813f710 0x7f8e981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:36156/0 (socket says 172.21.15.87:36156) 2024-03-25T18:21:22.695 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.694+0000 7f8e977fe640 1 -- 172.21.15.87:0/1302861630 learned_addr learned my addr 172.21.15.87:0/1302861630 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:22.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812419562 0 0) 0x7f8e980b2a30 con 0x7f8e9813f710 2024-03-25T18:21:22.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:21:22.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.710+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e80003520 con 0x7f8e9813f710 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573674550 0 0) 0x7f8e98142520 con 0x7f8e981430c0 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:21:22.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.716+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e980b2a30 con 0x7f8e981430c0 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854286491 0 0) 0x7f8e980b28a0 con 0x7f8e980a98f0 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:21:22.722 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.721+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e98142520 con 0x7f8e980a98f0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.731+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674832302 0 0) 0x7f8e980b2a30 con 0x7f8e981430c0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 >> v1:172.21.15.87:6790/0 conn(0x7f8e9813f710 legacy=0x7f8e981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 >> v1:172.21.15.171:6789/0 conn(0x7f8e980a98f0 legacy=0x7f8e980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 1 monclient: found mon.noname-a 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980b44c0 con 0x7f8e981430c0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e9e48f640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:21:22.733 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e900019f0 con 0x7f8e981430c0 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e96ffd640 10 monclient: dump: 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.734 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.732+0000 7f8e9e48f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e90001b50 con 0x7f8e981430c0 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 1 -- 172.21.15.87:0/1302861630 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e90001f60 con 0x7f8e981430c0 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e9e48f640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e9e48f640 4 set_mon_vals no callback set 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:22.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.748+0000 7f8e96ffd640 10 monclient: dump: 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:22.750 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 10 monclient: shutdown 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 1 -- 172.21.15.87:0/1302861630 >> v1:172.21.15.87:6789/0 conn(0x7f8e981430c0 legacy=0x7f8e9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 1 -- 172.21.15.87:0/1302861630 shutdown_connections 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 1 -- 172.21.15.87:0/1302861630 >> 172.21.15.87:0/1302861630 conn(0x7f8e980b7ea0 msgr2=0x7f8e980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:22.751 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 1 -- 172.21.15.87:0/1302861630 shutdown_connections 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.749+0000 7f8e9e48f640 1 -- 172.21.15.87:0/1302861630 wait complete. 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.750+0000 7f8e9e48f640 10 monclient: build_initial_monmap 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.750+0000 7f8e9e48f640 10 monclient: monmap: 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:22.752 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 librados: starting msgr at 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 librados: starting objecter 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 Processor -- start 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 -- start start 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 librados: setting wanted keys 2024-03-25T18:21:22.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 1 librados: calling monclient init 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.751+0000 7f8e9e48f640 10 monclient: init 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-b con 0x7f8e980a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient(hunting): picked mon.noname-a con 0x7f8e981430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e98139920 con 0x7f8e981430c0 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e981db380 con 0x7f8e980a98f0 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e981db620 con 0x7f8e9813f710 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:22.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.752+0000 7f8e9e48f640 10 monclient(hunting): authenticate will time out at 8198.047234s 2024-03-25T18:21:22.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.758+0000 7f8e97fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8e980a98f0 0x7f8e981dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:48778/0 (socket says 172.21.15.87:48778) 2024-03-25T18:21:22.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.758+0000 7f8e97fff640 1 -- 172.21.15.87:0/2005040109 learned_addr learned my addr 172.21.15.87:0/2005040109 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:22.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.760+0000 7f8e94ff9640 10 client.?.objecter ms_handle_connect 0x7f8e980a98f0 2024-03-25T18:21:22.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.760+0000 7f8e94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:22.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.767+0000 7f8e94ff9640 10 client.?.objecter ms_handle_connect 0x7f8e981430c0 2024-03-25T18:21:22.768 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.767+0000 7f8e94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.776+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761150499 0 0) 0x7f8e981db380 con 0x7f8e980a98f0 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.777+0000 7f8e94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.777+0000 7f8e94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.777+0000 7f8e94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.777+0000 7f8e94ff9640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:21:22.778 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.777+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e84004eb0 con 0x7f8e980a98f0 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722161943 0 0) 0x7f8e98139920 con 0x7f8e981430c0 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:21:22.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.782+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e981db380 con 0x7f8e981430c0 2024-03-25T18:21:22.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.784+0000 7f8e94ff9640 10 client.?.objecter ms_handle_connect 0x7f8e9813f710 2024-03-25T18:21:22.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.784+0000 7f8e94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 78941981 0 0) 0x7f8e84004eb0 con 0x7f8e980a98f0 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e98139920 con 0x7f8e980a98f0 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e88001a10 con 0x7f8e980a98f0 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:22.801 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:22.802 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:22.802 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:22.802 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:22.802 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.800+0000 7f8e94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:22.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2233970291 0 0) 0x7f8e981db380 con 0x7f8e981430c0 2024-03-25T18:21:22.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e84004eb0 con 0x7f8e981430c0 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e90001e60 con 0x7f8e981430c0 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.803+0000 7f8e94ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:22.805 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737513675 0 0) 0x7f8e981db620 con 0x7f8e9813f710 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:21:22.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.804+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e981db380 con 0x7f8e9813f710 2024-03-25T18:21:22.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 510094404 0 0) 0x7f8e98139920 con 0x7f8e980a98f0 2024-03-25T18:21:22.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 >> v1:172.21.15.87:6790/0 conn(0x7f8e9813f710 legacy=0x7f8e980b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:22.808 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 >> v1:172.21.15.87:6789/0 conn(0x7f8e981430c0 legacy=0x7f8e98137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 1 monclient: found mon.b 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980b0dc0 con 0x7f8e980a98f0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 10 monclient: _finish_auth 0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 10 monclient: _renew_subs 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e980b1010 con 0x7f8e980a98f0 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 10 monclient: _renew_subs 2024-03-25T18:21:22.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:22.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e980b0910 con 0x7f8e980a98f0 2024-03-25T18:21:22.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.807+0000 7f8e9e48f640 1 librados: init done 2024-03-25T18:21:22.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.808+0000 7f8e9e48f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:22.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.808+0000 7f8e9e48f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:22.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.808+0000 7f8e9e48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:22.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:22.808+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e980b0bb0 con 0x7f8e980a98f0 2024-03-25T18:21:23.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e88001c70 con 0x7f8e980a98f0 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f8e88003a60 con 0x7f8e980a98f0 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e94ff9640 10 monclient: dump: 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:23.091 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.089+0000 7f8e967fc640 4 set_mon_vals no callback set 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.090+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f8e88003c80 con 0x7f8e980a98f0 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.091+0000 7f8e977fe640 10 monclient: get_auth_request con 0x7f8e8406df50 auth_method 0 2024-03-25T18:21:23.092 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.091+0000 7f8e94ff9640 10 client.4528.objecter ms_handle_connect 0x7f8e8406df50 2024-03-25T18:21:23.095 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.094+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f8e88054880 con 0x7f8e980a98f0 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.094+0000 7f8e94ff9640 10 client.4528.objecter ms_dispatch 0x7f8e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.094+0000 7f8e94ff9640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.094+0000 7f8e94ff9640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.094+0000 7f8e94ff9640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.095+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e88050390 con 0x7f8e980a98f0 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.095+0000 7f8e94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:23.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.095+0000 7f8e94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:23.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.316+0000 7f8e9e48f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:23.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.316+0000 7f8e9e48f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:23.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.316+0000 7f8e9e48f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:23.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.316+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e980627b0 con 0x7f8e980a98f0 2024-03-25T18:21:23.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.347+0000 7f8e94ff9640 1 -- 172.21.15.87:0/2005040109 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f8e880522c0 con 0x7f8e980a98f0 2024-03-25T18:21:23.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.347+0000 7f8e94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:23.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.347+0000 7f8e94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:23.348 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:23.349 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:23.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.349+0000 7f8e9e48f640 10 librados: watch_flush enter 2024-03-25T18:21:23.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.349+0000 7f8e9e48f640 10 librados: watch_flush exit 2024-03-25T18:21:23.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.349+0000 7f8e9e48f640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.349+0000 7f8e9e48f640 10 client.4528.objecter successfully canceled tick 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.349+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 >> v1:172.21.15.171:6800/3417775508 conn(0x7f8e8406df50 legacy=0x7f8e84070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 10 monclient: shutdown 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 >> v1:172.21.15.171:6789/0 conn(0x7f8e980a98f0 legacy=0x7f8e981dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 shutdown_connections 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 >> 172.21.15.87:0/2005040109 conn(0x7f8e980b7ea0 msgr2=0x7f8e98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 shutdown_connections 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 -- 172.21.15.87:0/2005040109 wait complete. 2024-03-25T18:21:23.352 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:23.350+0000 7f8e9e48f640 1 librados: shutdown 2024-03-25T18:21:23.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:27.632 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:15.345 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.639 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:15.353 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.665 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:15.379 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.682 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:15.361 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.687 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:15.366 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.689 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:15.402 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.693 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:15.406 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.724 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:15.402 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:27.726 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:15.439 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:30.169 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.282+0000 7fc5e3765640 10 monclient: get_monmap_and_config 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.282+0000 7fc5e3765640 10 monclient: build_initial_monmap 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.282+0000 7fc5e3765640 10 monclient: monmap: 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:30.283446+0000 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:30.283446+0000 2024-03-25T18:21:30.286 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.283+0000 7fc5e3765640 1 Processor -- start 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.283+0000 7fc5e3765640 1 -- start start 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.283+0000 7fc5e3765640 10 monclient: init 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-c con 0x7fc5dc06ea40 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-b con 0x7fc5dc071800 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-a con 0x7fc5dc071c40 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc06f1f0 con 0x7fc5dc071c40 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc070d00 con 0x7fc5dc06ea40 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc0720b0 con 0x7fc5dc071800 2024-03-25T18:21:30.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:30.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.284+0000 7fc5e3765640 10 monclient(hunting): authenticate will time out at 8205.579323s 2024-03-25T18:21:30.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.289+0000 7fc5e1d02640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fc5dc071c40 0x7fc5dc14a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:56202/0 (socket says 172.21.15.87:56202) 2024-03-25T18:21:30.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.289+0000 7fc5e1d02640 1 -- 172.21.15.87:0/3135494255 learned_addr learned my addr 172.21.15.87:0/3135494255 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:30.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533910022 0 0) 0x7fc5dc0720b0 con 0x7fc5dc071800 2024-03-25T18:21:30.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:21:30.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.302+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c8003520 con 0x7fc5dc071800 2024-03-25T18:21:30.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426457230 0 0) 0x7fc5dc06f1f0 con 0x7fc5dc071c40 2024-03-25T18:21:30.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:21:30.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.305+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc0720b0 con 0x7fc5dc071c40 2024-03-25T18:21:30.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158975357 0 0) 0x7fc5dc070d00 con 0x7fc5dc06ea40 2024-03-25T18:21:30.311 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:21:30.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.310+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc06f1f0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2317446071 0 0) 0x7fc5dc06f1f0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 >> v1:172.21.15.87:6790/0 conn(0x7fc5dc071800 legacy=0x7fc5dc147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 >> v1:172.21.15.87:6789/0 conn(0x7fc5dc071c40 legacy=0x7fc5dc14a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 monclient: found mon.noname-c 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc072780 con 0x7fc5dc06ea40 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5d80015b0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5e3765640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5e3765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5d3fff640 10 monclient: dump: 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:30.318 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.319 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.315+0000 7fc5e3765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5d8001710 con 0x7fc5dc06ea40 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 1 -- 172.21.15.87:0/3135494255 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5d8001b60 con 0x7fc5dc06ea40 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5d3fff640 10 monclient: dump: 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:30.326 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5e3765640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:30.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.325+0000 7fc5e3765640 4 set_mon_vals no callback set 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 10 monclient: shutdown 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 1 -- 172.21.15.87:0/3135494255 >> v1:172.21.15.171:6789/0 conn(0x7fc5dc06ea40 legacy=0x7fc5dc13ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 1 -- 172.21.15.87:0/3135494255 shutdown_connections 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 1 -- 172.21.15.87:0/3135494255 >> 172.21.15.87:0/3135494255 conn(0x7fc5dc06b630 msgr2=0x7fc5dc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 1 -- 172.21.15.87:0/3135494255 shutdown_connections 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.326+0000 7fc5e3765640 1 -- 172.21.15.87:0/3135494255 wait complete. 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.327+0000 7fc5e3765640 10 monclient: build_initial_monmap 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.327+0000 7fc5e3765640 10 monclient: monmap: 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:30.329 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.327+0000 7fc5e3765640 1 librados: starting msgr at 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.327+0000 7fc5e3765640 1 librados: starting objecter 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.327+0000 7fc5e3765640 1 Processor -- start 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 -- start start 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 librados: setting wanted keys 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 librados: calling monclient init 2024-03-25T18:21:30.330 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient: init 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-b con 0x7fc5dc06ea40 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-c con 0x7fc5dc071800 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient(hunting): picked mon.noname-a con 0x7fc5dc071c40 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc133160 con 0x7fc5dc071c40 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc131740 con 0x7fc5dc06ea40 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc133440 con 0x7fc5dc071800 2024-03-25T18:21:30.331 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:30.332 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.328+0000 7fc5e3765640 10 monclient(hunting): authenticate will time out at 8205.623583s 2024-03-25T18:21:30.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.333+0000 7fc5e0d00640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fc5dc071800 0x7fc5dc131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:49094/0 (socket says 172.21.15.87:49094) 2024-03-25T18:21:30.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.333+0000 7fc5e0d00640 1 -- 172.21.15.87:0/3068124919 learned_addr learned my addr 172.21.15.87:0/3068124919 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:30.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.335+0000 7fc5d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5dc071800 2024-03-25T18:21:30.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.335+0000 7fc5d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:30.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.336+0000 7fc5d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5dc071c40 2024-03-25T18:21:30.337 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.336+0000 7fc5d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:30.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.337+0000 7fc5d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5dc06ea40 2024-03-25T18:21:30.338 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.337+0000 7fc5d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561856805 0 0) 0x7fc5dc133440 con 0x7fc5dc071800 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:21:30.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.341+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c0004eb0 con 0x7fc5dc071800 2024-03-25T18:21:30.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117246260 0 0) 0x7fc5dc131740 con 0x7fc5dc06ea40 2024-03-25T18:21:30.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:21:30.347 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.345+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc133440 con 0x7fc5dc06ea40 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462461026 0 0) 0x7fc5dc133160 con 0x7fc5dc071c40 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:21:30.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.350+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc131740 con 0x7fc5dc071c40 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000758977 0 0) 0x7fc5c0004eb0 con 0x7fc5dc071800 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5dc133160 con 0x7fc5dc071800 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5cc001990 con 0x7fc5dc071800 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.357 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.355+0000 7fc5d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.356+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 32300570 0 0) 0x7fc5dc133440 con 0x7fc5dc06ea40 2024-03-25T18:21:30.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.356+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c0004eb0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.359+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5d8001a40 con 0x7fc5dc06ea40 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.359+0000 7fc5d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.359+0000 7fc5d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.359+0000 7fc5d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:30.360 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:30.361 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1276984186 0 0) 0x7fc5c0004eb0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 >> v1:172.21.15.87:6790/0 conn(0x7fc5dc071800 legacy=0x7fc5dc131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 >> v1:172.21.15.87:6789/0 conn(0x7fc5dc071c40 legacy=0x7fc5dc1e6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 1 monclient: found mon.b 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc132370 con 0x7fc5dc06ea40 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 10 monclient: _renew_subs 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc132650 con 0x7fc5dc06ea40 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 10 monclient: _renew_subs 2024-03-25T18:21:30.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc11d0f0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.366+0000 7fc5e3765640 1 librados: init done 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.367+0000 7fc5e3765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.367+0000 7fc5e3765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.367+0000 7fc5e3765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.367+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc1318d0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5d8001ba0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc5d8003a40 con 0x7fc5dc06ea40 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d1ffb640 10 monclient: dump: 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:30.652 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:30.653 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:30.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.650+0000 7fc5d37fe640 4 set_mon_vals no callback set 2024-03-25T18:21:30.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.651+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fc5d8019a10 con 0x7fc5dc06ea40 2024-03-25T18:21:30.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.652+0000 7fc5e0d00640 10 monclient: get_auth_request con 0x7fc5c006df40 auth_method 0 2024-03-25T18:21:30.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.652+0000 7fc5d1ffb640 10 client.4534.objecter ms_handle_connect 0x7fc5c006df40 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fc5d80536b0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 10 client.4534.objecter ms_dispatch 0x7fc5dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:21:30.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5d804eff0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.659 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:30.659 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.657+0000 7fc5d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:30.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.885+0000 7fc5e3765640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:30.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.885+0000 7fc5e3765640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:30.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.885+0000 7fc5e3765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:30.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.885+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5dc0627b0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.911+0000 7fc5d1ffb640 1 -- 172.21.15.87:0/3068124919 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fc5d80510f0 con 0x7fc5dc06ea40 2024-03-25T18:21:30.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.911+0000 7fc5d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:30.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.911+0000 7fc5d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:30.912 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:30.912 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.913+0000 7fc5e3765640 10 librados: watch_flush enter 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 10 librados: watch_flush exit 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 >> v1:172.21.15.171:6800/3417775508 conn(0x7fc5c006df40 legacy=0x7fc5c0070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 10 monclient: shutdown 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 >> v1:172.21.15.171:6789/0 conn(0x7fc5dc06ea40 legacy=0x7fc5dc12bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 shutdown_connections 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 >> 172.21.15.87:0/3068124919 conn(0x7fc5dc06b630 msgr2=0x7fc5dc0af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 shutdown_connections 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 -- 172.21.15.87:0/3068124919 wait complete. 2024-03-25T18:21:30.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:30.914+0000 7fc5e3765640 1 librados: shutdown 2024-03-25T18:21:30.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:37.735 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.846+0000 7f623064a640 10 monclient: get_monmap_and_config 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.846+0000 7f623064a640 10 monclient: build_initial_monmap 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.846+0000 7f623064a640 10 monclient: monmap: 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:37.849 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:37.848762+0000 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:37.848762+0000 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.847+0000 7f623064a640 1 Processor -- start 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.847+0000 7f623064a640 1 -- start start 2024-03-25T18:21:37.850 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.847+0000 7f623064a640 10 monclient: init 2024-03-25T18:21:37.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:37.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient(hunting): picked mon.noname-a con 0x7f62280a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient(hunting): picked mon.noname-b con 0x7f622813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient(hunting): picked mon.noname-c con 0x7f62281430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6228142520 con 0x7f62280a98f0 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62280b28a0 con 0x7f62281430c0 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62280b2a30 con 0x7f622813f710 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:37.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.848+0000 7f623064a640 10 monclient(hunting): authenticate will time out at 8213.144482s 2024-03-25T18:21:37.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.857+0000 7f622dbe5640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f622813f710 0x7f62281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:34628/0 (socket says 172.21.15.87:34628) 2024-03-25T18:21:37.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.857+0000 7f622dbe5640 1 -- 172.21.15.87:0/3736332944 learned_addr learned my addr 172.21.15.87:0/3736332944 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074608454 0 0) 0x7f62280b2a30 con 0x7f622813f710 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 10 monclient(hunting): _negotiate 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:21:37.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.870+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6214003520 con 0x7f622813f710 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299883047 0 0) 0x7f62280b28a0 con 0x7f62281430c0 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 10 monclient(hunting): _negotiate 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:21:37.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.871+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62280b2a30 con 0x7f62281430c0 2024-03-25T18:21:37.880 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216438291 0 0) 0x7f6228142520 con 0x7f62280a98f0 2024-03-25T18:21:37.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 10 monclient(hunting): _negotiate 2024-03-25T18:21:37.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:37.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:37.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:21:37.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.879+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62280b28a0 con 0x7f62280a98f0 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.889+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621029740 0 0) 0x7f6214003520 con 0x7f622813f710 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.889+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 >> v1:172.21.15.171:6789/0 conn(0x7f62281430c0 legacy=0x7f622814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 >> v1:172.21.15.87:6789/0 conn(0x7f62280a98f0 legacy=0x7f62280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: _finish_hunting 0 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 1 monclient: found mon.noname-b 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:21:37.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62280b44c0 con 0x7f622813f710 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: _finish_auth 0 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: _check_auth_tickets 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6218001690 con 0x7f622813f710 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f622d3e4640 10 monclient: dump: 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:37.893 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:37.894 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:37.894 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:37.894 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f623064a640 5 monclient: authenticate success, global_id 4541 2024-03-25T18:21:37.894 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.890+0000 7f623064a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62180017f0 con 0x7f622813f710 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 1 -- 172.21.15.87:0/3736332944 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6218001c00 con 0x7f622813f710 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f623064a640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f623064a640 4 set_mon_vals no callback set 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.917+0000 7f622d3e4640 10 monclient: dump: 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:37.919 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:37.920 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 10 monclient: shutdown 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 1 -- 172.21.15.87:0/3736332944 >> v1:172.21.15.87:6790/0 conn(0x7f622813f710 legacy=0x7f62281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 1 -- 172.21.15.87:0/3736332944 shutdown_connections 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 1 -- 172.21.15.87:0/3736332944 >> 172.21.15.87:0/3736332944 conn(0x7f62280b7ea0 msgr2=0x7f62280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:37.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 1 -- 172.21.15.87:0/3736332944 shutdown_connections 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.918+0000 7f623064a640 1 -- 172.21.15.87:0/3736332944 wait complete. 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 10 monclient: build_initial_monmap 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 10 monclient: monmap: 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:37.923 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 librados: starting msgr at 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 librados: starting objecter 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 Processor -- start 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 -- start start 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 librados: setting wanted keys 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 1 librados: calling monclient init 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.919+0000 7f623064a640 10 monclient: init 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient(hunting): picked mon.noname-c con 0x7f62280a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient(hunting): picked mon.noname-b con 0x7f622813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient(hunting): picked mon.noname-a con 0x7f62281430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:37.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62280aba00 con 0x7f62281430c0 2024-03-25T18:21:37.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62281d7490 con 0x7f622813f710 2024-03-25T18:21:37.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62281d8360 con 0x7f62280a98f0 2024-03-25T18:21:37.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:37.925 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.920+0000 7f623064a640 10 monclient(hunting): authenticate will time out at 8213.216226s 2024-03-25T18:21:37.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.936+0000 7f622dbe5640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f622813f710 0x7f62281d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:57146/0 (socket says 172.21.15.87:57146) 2024-03-25T18:21:37.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.936+0000 7f622dbe5640 1 -- 172.21.15.87:0/1481333764 learned_addr learned my addr 172.21.15.87:0/1481333764 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:37.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.937+0000 7f621effd640 10 client.?.objecter ms_handle_connect 0x7f622813f710 2024-03-25T18:21:37.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.937+0000 7f621effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:37.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367701329 0 0) 0x7f62281d7490 con 0x7f622813f710 2024-03-25T18:21:37.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:37.951 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:37.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:37.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:21:37.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.949+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c004eb0 con 0x7f622813f710 2024-03-25T18:21:37.955 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.953+0000 7f621effd640 10 client.?.objecter ms_handle_connect 0x7f62281430c0 2024-03-25T18:21:37.955 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.953+0000 7f621effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:37.962 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.960+0000 7f621effd640 10 client.?.objecter ms_handle_connect 0x7f62280a98f0 2024-03-25T18:21:37.962 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.960+0000 7f621effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544183897 0 0) 0x7f62280aba00 con 0x7f62281430c0 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:21:37.967 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.965+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62281d7490 con 0x7f62281430c0 2024-03-25T18:21:37.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304861852 0 0) 0x7f620c004eb0 con 0x7f622813f710 2024-03-25T18:21:37.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62280aba00 con 0x7f622813f710 2024-03-25T18:21:37.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6218001bc0 con 0x7f622813f710 2024-03-25T18:21:37.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 10 monclient(hunting): dump: 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:37.970 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:37.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.967+0000 7f621effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895922960 0 0) 0x7f62280aba00 con 0x7f622813f710 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 >> v1:172.21.15.87:6790/0 conn(0x7f62280a98f0 legacy=0x7f62281386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 >> v1:172.21.15.87:6789/0 conn(0x7f62281430c0 legacy=0x7f6228133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 1 monclient: found mon.b 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:37.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62280abc30 con 0x7f622813f710 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 10 monclient: _finish_auth 0 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f621effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 10 monclient: _renew_subs 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6228137940 con 0x7f622813f710 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 10 monclient: _renew_subs 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6228135220 con 0x7f622813f710 2024-03-25T18:21:37.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.975+0000 7f623064a640 1 librados: init done 2024-03-25T18:21:37.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.976+0000 7f623064a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:37.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.976+0000 7f623064a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:37.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.976+0000 7f623064a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:37.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:37.976+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62280627b0 con 0x7f622813f710 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62180011f0 con 0x7f622813f710 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6218004cb0 con 0x7f622813f710 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:38.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f621effd640 10 monclient: dump: 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:38.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.262+0000 7f622cbe3640 4 set_mon_vals no callback set 2024-03-25T18:21:38.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.263+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f621801ad00 con 0x7f622813f710 2024-03-25T18:21:38.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.263+0000 7f622e3e6640 10 monclient: get_auth_request con 0x7f620c06dd00 auth_method 0 2024-03-25T18:21:38.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.263+0000 7f621effd640 10 client.4540.objecter ms_handle_connect 0x7f620c06dd00 2024-03-25T18:21:38.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.268+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f62180549d0 con 0x7f622813f710 2024-03-25T18:21:38.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.268+0000 7f621effd640 10 client.4540.objecter ms_dispatch 0x7f6228000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:38.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.268+0000 7f621effd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:38.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.268+0000 7f621effd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:38.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.268+0000 7f621effd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:21:38.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.269+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6218050310 con 0x7f622813f710 2024-03-25T18:21:38.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.269+0000 7f621effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:38.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.269+0000 7f621effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:38.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.488+0000 7f623064a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:38.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.488+0000 7f623064a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:38.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.488+0000 7f623064a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:38.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.488+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62281d7830 con 0x7f622813f710 2024-03-25T18:21:38.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.510+0000 7f621effd640 1 -- 172.21.15.87:0/1481333764 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f6218052410 con 0x7f622813f710 2024-03-25T18:21:38.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.510+0000 7f621effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:38.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.510+0000 7f621effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:38.513 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:38.513 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 10 librados: watch_flush enter 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 10 librados: watch_flush exit 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 >> v1:172.21.15.171:6800/3417775508 conn(0x7f620c06dd00 legacy=0x7f620c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 10 monclient: shutdown 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 >> v1:172.21.15.171:6789/0 conn(0x7f622813f710 legacy=0x7f62281d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 shutdown_connections 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 >> 172.21.15.87:0/1481333764 conn(0x7f62280b7ea0 msgr2=0x7f6228074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 shutdown_connections 2024-03-25T18:21:38.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.513+0000 7f623064a640 1 -- 172.21.15.87:0/1481333764 wait complete. 2024-03-25T18:21:38.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:38.514+0000 7f623064a640 1 librados: shutdown 2024-03-25T18:21:38.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:45.334 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:45.452 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.451+0000 7fab56d1c640 10 monclient: get_monmap_and_config 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.451+0000 7fab56d1c640 10 monclient: build_initial_monmap 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.452+0000 7fab56d1c640 10 monclient: monmap: 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:45.452829+0000 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:45.452829+0000 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:45.453 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.452+0000 7fab56d1c640 1 Processor -- start 2024-03-25T18:21:45.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 1 -- start start 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient: init 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fab500b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fab500b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fab5014be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500af940 con 0x7fab500b4d60 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500b0a70 con 0x7fab5014be90 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500b0d50 con 0x7fab500b1730 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:45.455 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.453+0000 7fab56d1c640 10 monclient(hunting): authenticate will time out at 8220.748573s 2024-03-25T18:21:45.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.459+0000 7fab552b9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab5014be90 0x7fab5014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:33846/0 (socket says 172.21.15.87:33846) 2024-03-25T18:21:45.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.459+0000 7fab552b9640 1 -- 172.21.15.87:0/3024883945 learned_addr learned my addr 172.21.15.87:0/3024883945 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:45.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792889391 0 0) 0x7fab500b0a70 con 0x7fab5014be90 2024-03-25T18:21:45.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:45.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:45.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:45.477 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:21:45.478 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.476+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40003520 con 0x7fab5014be90 2024-03-25T18:21:45.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867425085 0 0) 0x7fab500b0d50 con 0x7fab500b1730 2024-03-25T18:21:45.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:45.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:45.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:45.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:21:45.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.479+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab500b0a70 con 0x7fab500b1730 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231120221 0 0) 0x7fab500b0a70 con 0x7fab500b1730 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 >> v1:172.21.15.171:6789/0 conn(0x7fab5014be90 legacy=0x7fab5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 >> v1:172.21.15.87:6789/0 conn(0x7fab500b4d60 legacy=0x7fab5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:45.489 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500b1030 con 0x7fab500b1730 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fab3c001670 con 0x7fab500b1730 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab56d1c640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab56d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab4f7fe640 10 monclient: dump: 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.490 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.488+0000 7fab56d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab3c0017d0 con 0x7fab500b1730 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 1 -- 172.21.15.87:0/3024883945 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fab3c001ba0 con 0x7fab500b1730 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.504+0000 7fab4f7fe640 10 monclient: dump: 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.506 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:45.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 4 set_mon_vals no callback set 2024-03-25T18:21:45.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 10 monclient: shutdown 2024-03-25T18:21:45.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:45.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 1 -- 172.21.15.87:0/3024883945 >> v1:172.21.15.87:6790/0 conn(0x7fab500b1730 legacy=0x7fab500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:45.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 1 -- 172.21.15.87:0/3024883945 shutdown_connections 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 1 -- 172.21.15.87:0/3024883945 >> 172.21.15.87:0/3024883945 conn(0x7fab500617a0 msgr2=0x7fab50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 1 -- 172.21.15.87:0/3024883945 shutdown_connections 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.505+0000 7fab56d1c640 1 -- 172.21.15.87:0/3024883945 wait complete. 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.506+0000 7fab56d1c640 10 monclient: build_initial_monmap 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.506+0000 7fab56d1c640 10 monclient: monmap: 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.506+0000 7fab56d1c640 1 librados: starting msgr at 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.506+0000 7fab56d1c640 1 librados: starting objecter 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 1 Processor -- start 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 1 -- start start 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 1 librados: setting wanted keys 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 1 librados: calling monclient init 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient: init 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fab500b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fab500b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.507+0000 7fab56d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fab5014be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.508+0000 7fab56d1c640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50141d30 con 0x7fab5014be90 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.508+0000 7fab56d1c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50140a30 con 0x7fab500b1730 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.508+0000 7fab56d1c640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab50140ec0 con 0x7fab500b4d60 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.508+0000 7fab56d1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:45.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.508+0000 7fab56d1c640 10 monclient(hunting): authenticate will time out at 8220.803021s 2024-03-25T18:21:45.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.517+0000 7fab54ab8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab500b1730 0x7fab50131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:33906/0 (socket says 172.21.15.87:33906) 2024-03-25T18:21:45.518 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.517+0000 7fab54ab8640 1 -- 172.21.15.87:0/1711639617 learned_addr learned my addr 172.21.15.87:0/1711639617 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:45.520 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.519+0000 7fab4d7fa640 10 client.?.objecter ms_handle_connect 0x7fab500b1730 2024-03-25T18:21:45.521 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.520+0000 7fab4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886759332 0 0) 0x7fab50140a30 con 0x7fab500b1730 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:21:45.525 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.524+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab500b1730 2024-03-25T18:21:45.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.527+0000 7fab4d7fa640 10 client.?.objecter ms_handle_connect 0x7fab500b4d60 2024-03-25T18:21:45.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.527+0000 7fab4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:45.534 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268499004 0 0) 0x7fab34004eb0 con 0x7fab500b1730 2024-03-25T18:21:45.534 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab50140a30 con 0x7fab500b1730 2024-03-25T18:21:45.534 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fab44001a30 con 0x7fab500b1730 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.533+0000 7fab4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:45.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.534+0000 7fab4d7fa640 10 client.?.objecter ms_handle_connect 0x7fab5014be90 2024-03-25T18:21:45.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.534+0000 7fab4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:45.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162231988 0 0) 0x7fab50140ec0 con 0x7fab500b4d60 2024-03-25T18:21:45.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:45.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:45.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:45.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:21:45.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.541+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab500b4d60 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2300744238 0 0) 0x7fab50140a30 con 0x7fab500b1730 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 >> v1:172.21.15.87:6790/0 conn(0x7fab500b4d60 legacy=0x7fab501dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 >> v1:172.21.15.87:6789/0 conn(0x7fab5014be90 legacy=0x7fab501e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 1 monclient: found mon.b 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab50137f40 con 0x7fab500b1730 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:21:45.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 10 monclient: _renew_subs 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab501381f0 con 0x7fab500b1730 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 10 monclient: _renew_subs 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50132690 con 0x7fab500b1730 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.543+0000 7fab56d1c640 1 librados: init done 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.544+0000 7fab56d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.544+0000 7fab56d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.544+0000 7fab56d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:45.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.544+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab500627b0 con 0x7fab500b1730 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab44001cf0 con 0x7fab500b1730 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fab44003a60 con 0x7fab500b1730 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4d7fa640 10 monclient: dump: 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.816 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.815+0000 7fab4effd640 4 set_mon_vals no callback set 2024-03-25T18:21:45.817 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.816+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fab44003c80 con 0x7fab500b1730 2024-03-25T18:21:45.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.816+0000 7fab4ffff640 10 monclient: get_auth_request con 0x7fab3406dd40 auth_method 0 2024-03-25T18:21:45.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.816+0000 7fab4d7fa640 10 client.4546.objecter ms_handle_connect 0x7fab3406dd40 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fab44054910 con 0x7fab500b1730 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 10 client.4546.objecter ms_dispatch 0x7fab50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab44050250 con 0x7fab500b1730 2024-03-25T18:21:45.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:45.823 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:45.821+0000 7fab4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:46.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.052+0000 7fab56d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:46.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.052+0000 7fab56d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:46.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.052+0000 7fab56d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:21:46.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.052+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab50069f40 con 0x7fab500b1730 2024-03-25T18:21:46.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.076+0000 7fab4d7fa640 1 -- 172.21.15.87:0/1711639617 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fab44052350 con 0x7fab500b1730 2024-03-25T18:21:46.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.076+0000 7fab4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:46.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.076+0000 7fab4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:46.078 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:46.078 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:46.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 10 librados: watch_flush enter 2024-03-25T18:21:46.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 10 librados: watch_flush exit 2024-03-25T18:21:46.080 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 >> v1:172.21.15.171:6800/3417775508 conn(0x7fab3406dd40 legacy=0x7fab34070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 10 monclient: shutdown 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.078+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 >> v1:172.21.15.171:6789/0 conn(0x7fab500b1730 legacy=0x7fab50131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.079+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 shutdown_connections 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.079+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 >> 172.21.15.87:0/1711639617 conn(0x7fab500617a0 msgr2=0x7fab500b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.079+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 shutdown_connections 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.079+0000 7fab56d1c640 1 -- 172.21.15.87:0/1711639617 wait complete. 2024-03-25T18:21:46.081 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:46.079+0000 7fab56d1c640 1 librados: shutdown 2024-03-25T18:21:46.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:47.666 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:35.379 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.671 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:35.384 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.696 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:35.409 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.734 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:35.413 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.737 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:35.450 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.739 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:35.418 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.740 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:35.453 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.772 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:35.451 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:47.777 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:35.490 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.899 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.011+0000 7f446976d640 10 monclient: get_monmap_and_config 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.011+0000 7f446976d640 10 monclient: build_initial_monmap 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.011+0000 7f446976d640 10 monclient: monmap: 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.012 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:21:53.012191+0000 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:21:53.012191+0000 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:21:53.013 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 1 Processor -- start 2024-03-25T18:21:53.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 1 -- start start 2024-03-25T18:21:53.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 10 monclient: init 2024-03-25T18:21:53.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 10 monclient(hunting): picked mon.noname-b con 0x7f44640a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.012+0000 7f446976d640 10 monclient(hunting): picked mon.noname-a con 0x7f446413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 10 monclient(hunting): picked mon.noname-c con 0x7f44641430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4464142520 con 0x7f446413f710 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44640b28a0 con 0x7f44641430c0 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44640b2a30 con 0x7f44640a98f0 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:53.015 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.013+0000 7f446976d640 10 monclient(hunting): authenticate will time out at 8228.308019s 2024-03-25T18:21:53.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.017+0000 7f4462ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f44640a98f0 0x7f44640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55738/0 (socket says 172.21.15.87:55738) 2024-03-25T18:21:53.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.017+0000 7f4462ffd640 1 -- 172.21.15.87:0/1081441406 learned_addr learned my addr 172.21.15.87:0/1081441406 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527122823 0 0) 0x7f44640b2a30 con 0x7f44640a98f0 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:21:53.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.026+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450003520 con 0x7f44640a98f0 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236026591 0 0) 0x7f44640b28a0 con 0x7f44641430c0 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:21:53.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.036+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44640b2a30 con 0x7f44641430c0 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378237287 0 0) 0x7f4464142520 con 0x7f446413f710 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:21:53.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.038+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44640b28a0 con 0x7f446413f710 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742513008 0 0) 0x7f4450003520 con 0x7f44640a98f0 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 >> v1:172.21.15.171:6789/0 conn(0x7f44641430c0 legacy=0x7f446414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 >> v1:172.21.15.87:6789/0 conn(0x7f446413f710 legacy=0x7f44641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 monclient: found mon.noname-b 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:53.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44640b44c0 con 0x7f44640a98f0 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4454001690 con 0x7f44640a98f0 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f446976d640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f446976d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f44627fc640 10 monclient: dump: 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.042 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.039+0000 7f446976d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44540017f0 con 0x7f44640a98f0 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 1 -- 172.21.15.87:0/1081441406 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4454001c00 con 0x7f44640a98f0 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 4 set_mon_vals no callback set 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f44627fc640 10 monclient: dump: 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.057 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:53.058 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:53.058 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:53.058 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 10 monclient: shutdown 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 1 -- 172.21.15.87:0/1081441406 >> v1:172.21.15.87:6790/0 conn(0x7f44640a98f0 legacy=0x7f44640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 1 -- 172.21.15.87:0/1081441406 shutdown_connections 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 1 -- 172.21.15.87:0/1081441406 >> 172.21.15.87:0/1081441406 conn(0x7f44640b7ea0 msgr2=0x7f44640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 1 -- 172.21.15.87:0/1081441406 shutdown_connections 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.056+0000 7f446976d640 1 -- 172.21.15.87:0/1081441406 wait complete. 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.057+0000 7f446976d640 10 monclient: build_initial_monmap 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.057+0000 7f446976d640 10 monclient: monmap: 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:21:53.060 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.057+0000 7f446976d640 1 librados: starting msgr at 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.057+0000 7f446976d640 1 librados: starting objecter 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 1 Processor -- start 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 1 -- start start 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 1 librados: setting wanted keys 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 1 librados: calling monclient init 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient: init 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:53.061 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient(hunting): picked mon.noname-c con 0x7f44640a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient(hunting): picked mon.noname-b con 0x7f446413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 10 monclient(hunting): picked mon.noname-a con 0x7f44641430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.058+0000 7f446976d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44640abd30 con 0x7f44641430c0 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.059+0000 7f446976d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44640abfa0 con 0x7f446413f710 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.059+0000 7f446976d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44641d77a0 con 0x7f44640a98f0 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.059+0000 7f446976d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:53.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.059+0000 7f446976d640 10 monclient(hunting): authenticate will time out at 8228.353930s 2024-03-25T18:21:53.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.077+0000 7f4462ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f44640a98f0 0x7f44641d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:55762/0 (socket says 172.21.15.87:55762) 2024-03-25T18:21:53.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.077+0000 7f4462ffd640 1 -- 172.21.15.87:0/1642792461 learned_addr learned my addr 172.21.15.87:0/1642792461 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:21:53.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.078+0000 7f445bfff640 10 client.?.objecter ms_handle_connect 0x7f44640a98f0 2024-03-25T18:21:53.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.078+0000 7f445bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:53.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.083+0000 7f445bfff640 10 client.?.objecter ms_handle_connect 0x7f446413f710 2024-03-25T18:21:53.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.083+0000 7f445bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:53.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.083+0000 7f445bfff640 10 client.?.objecter ms_handle_connect 0x7f44641430c0 2024-03-25T18:21:53.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.083+0000 7f445bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846847464 0 0) 0x7f44640abd30 con 0x7f44641430c0 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4444004eb0 con 0x7f44641430c0 2024-03-25T18:21:53.090 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557673611 0 0) 0x7f44641d77a0 con 0x7f44640a98f0 2024-03-25T18:21:53.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:21:53.091 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.089+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44640abd30 con 0x7f44640a98f0 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.096+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221911969 0 0) 0x7f4444004eb0 con 0x7f44641430c0 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44641d77a0 con 0x7f44641430c0 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f445c001bd0 con 0x7f44641430c0 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.098 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.097+0000 7f445bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838991339 0 0) 0x7f44640abfa0 con 0x7f446413f710 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:21:53.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.098+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4444004eb0 con 0x7f446413f710 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184470396 0 0) 0x7f44640abd30 con 0x7f44640a98f0 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44640abfa0 con 0x7f44640a98f0 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4454001b40 con 0x7f44640a98f0 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.105+0000 7f445bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:53.107 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:53.108 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754369774 0 0) 0x7f44640abfa0 con 0x7f44640a98f0 2024-03-25T18:21:53.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 >> v1:172.21.15.171:6789/0 conn(0x7f446413f710 legacy=0x7f44641d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 >> v1:172.21.15.87:6789/0 conn(0x7f44641430c0 legacy=0x7f44641385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:53.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 1 monclient: found mon.c 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44641d83d0 con 0x7f44640a98f0 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f445bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 10 monclient: _renew_subs 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44641d41e0 con 0x7f44640a98f0 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 10 monclient: _renew_subs 2024-03-25T18:21:53.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44641d5110 con 0x7f44640a98f0 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.112+0000 7f446976d640 1 librados: init done 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.113+0000 7f446976d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.113+0000 7f446976d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.113+0000 7f446976d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.113+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44640627b0 con 0x7f44640a98f0 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44540011f0 con 0x7f44640a98f0 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4454004c30 con 0x7f44640a98f0 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f445bfff640 10 monclient: dump: 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:53.431 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.429+0000 7f4461ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:53.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.430+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f445401ac80 con 0x7f44640a98f0 2024-03-25T18:21:53.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.431+0000 7f445a7fc640 10 monclient: get_auth_request con 0x7f444406df20 auth_method 0 2024-03-25T18:21:53.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.432+0000 7f445bfff640 10 client.4556.objecter ms_handle_connect 0x7f444406df20 2024-03-25T18:21:53.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4454054900 con 0x7f44640a98f0 2024-03-25T18:21:53.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 10 client.4556.objecter ms_dispatch 0x7f4464000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:53.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:53.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:53.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T18:21:53.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4454050240 con 0x7f44640a98f0 2024-03-25T18:21:53.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:53.440 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.438+0000 7f445bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:53.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.665+0000 7f446976d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:53.666 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.665+0000 7f446976d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:53.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.665+0000 7f446976d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:21:53.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.665+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44641d8560 con 0x7f44640a98f0 2024-03-25T18:21:53.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.714+0000 7f445bfff640 1 -- 172.21.15.87:0/1642792461 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4454052340 con 0x7f44640a98f0 2024-03-25T18:21:53.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.714+0000 7f445bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:53.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.714+0000 7f445bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:53.715 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:21:53.715 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:53.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 10 librados: watch_flush enter 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 10 librados: watch_flush exit 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 10 client.4556.objecter successfully canceled tick 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 >> v1:172.21.15.171:6800/3417775508 conn(0x7f444406df20 legacy=0x7f44440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 10 monclient: shutdown 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 >> v1:172.21.15.87:6790/0 conn(0x7f44640a98f0 legacy=0x7f44641d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 shutdown_connections 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 >> 172.21.15.87:0/1642792461 conn(0x7f44640b7ea0 msgr2=0x7f446413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 shutdown_connections 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 -- 172.21.15.87:0/1642792461 wait complete. 2024-03-25T18:21:53.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:21:53.717+0000 7f446976d640 1 librados: shutdown 2024-03-25T18:21:53.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:00.538 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.650+0000 7f78bc715640 10 monclient: get_monmap_and_config 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.650+0000 7f78bc715640 10 monclient: build_initial_monmap 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.650+0000 7f78bc715640 10 monclient: monmap: 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:00.651210+0000 2024-03-25T18:22:00.651 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:00.651210+0000 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.651+0000 7f78bc715640 1 Processor -- start 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.651+0000 7f78bc715640 1 -- start start 2024-03-25T18:22:00.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.651+0000 7f78bc715640 10 monclient: init 2024-03-25T18:22:00.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:00.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-b con 0x7f78b406ab80 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-a con 0x7f78b406af60 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-c con 0x7f78b40b00b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40b0500 con 0x7f78b406af60 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b413ffa0 con 0x7f78b40b00b0 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40af520 con 0x7f78b406ab80 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:00.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.652+0000 7f78bc715640 10 monclient(hunting): authenticate will time out at 8235.947284s 2024-03-25T18:22:00.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.653+0000 7f78bacb2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78b40b00b0 0x7f78b414dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:39900/0 (socket says 172.21.15.87:39900) 2024-03-25T18:22:00.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.653+0000 7f78bacb2640 1 -- 172.21.15.87:0/2930462485 learned_addr learned my addr 172.21.15.87:0/2930462485 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118001928 0 0) 0x7f78b413ffa0 con 0x7f78b40b00b0 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 10 monclient(hunting): _negotiate 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:22:00.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.673+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a0003520 con 0x7f78b40b00b0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856930670 0 0) 0x7f78a0003520 con 0x7f78b40b00b0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 >> v1:172.21.15.87:6790/0 conn(0x7f78b406ab80 legacy=0x7f78b40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 >> v1:172.21.15.87:6789/0 conn(0x7f78b406af60 legacy=0x7f78b413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: _finish_hunting 0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 monclient: found mon.noname-c 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4072890 con 0x7f78b40b00b0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: _finish_auth 0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: _check_auth_tickets 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f78ac001a10 con 0x7f78b40b00b0 2024-03-25T18:22:00.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78bc715640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78bc715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78b94af640 10 monclient: dump: 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.679+0000 7f78bc715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:00.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78ac001b70 con 0x7f78b40b00b0 2024-03-25T18:22:00.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 1 -- 172.21.15.87:0/2930462485 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f78ac001f80 con 0x7f78b40b00b0 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78bc715640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78bc715640 4 set_mon_vals no callback set 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.684+0000 7f78b94af640 10 monclient: dump: 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:00.686 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:00.687 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:00.687 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:00.687 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 10 monclient: shutdown 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 1 -- 172.21.15.87:0/2930462485 >> v1:172.21.15.171:6789/0 conn(0x7f78b40b00b0 legacy=0x7f78b414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 1 -- 172.21.15.87:0/2930462485 shutdown_connections 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 1 -- 172.21.15.87:0/2930462485 >> 172.21.15.87:0/2930462485 conn(0x7f78b40b4a80 msgr2=0x7f78b40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.685+0000 7f78bc715640 1 -- 172.21.15.87:0/2930462485 shutdown_connections 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.686+0000 7f78bc715640 1 -- 172.21.15.87:0/2930462485 wait complete. 2024-03-25T18:22:00.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.686+0000 7f78bc715640 10 monclient: build_initial_monmap 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.686+0000 7f78bc715640 10 monclient: monmap: 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 librados: starting msgr at 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 librados: starting objecter 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 Processor -- start 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 -- start start 2024-03-25T18:22:00.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 librados: setting wanted keys 2024-03-25T18:22:00.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 1 librados: calling monclient init 2024-03-25T18:22:00.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.687+0000 7f78bc715640 10 monclient: init 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-c con 0x7f78b406ab80 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-b con 0x7f78b40b00b0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient(hunting): picked mon.noname-a con 0x7f78b4139a10 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b413ad00 con 0x7f78b4139a10 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b41d4f20 con 0x7f78b40b00b0 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b413b230 con 0x7f78b406ab80 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:00.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.688+0000 7f78bc715640 10 monclient(hunting): authenticate will time out at 8235.983057s 2024-03-25T18:22:00.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.690+0000 7f78b9cb0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78b40b00b0 0x7f78b41da8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:39928/0 (socket says 172.21.15.87:39928) 2024-03-25T18:22:00.692 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.690+0000 7f78b9cb0640 1 -- 172.21.15.87:0/148600660 learned_addr learned my addr 172.21.15.87:0/148600660 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:00.696 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.695+0000 7f78aaffd640 10 client.?.objecter ms_handle_connect 0x7f78b40b00b0 2024-03-25T18:22:00.697 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.695+0000 7f78aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517450206 0 0) 0x7f78b41d4f20 con 0x7f78b40b00b0 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:22:00.700 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.698+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7898004eb0 con 0x7f78b40b00b0 2024-03-25T18:22:00.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1446427885 0 0) 0x7f7898004eb0 con 0x7f78b40b00b0 2024-03-25T18:22:00.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78b41d4f20 con 0x7f78b40b00b0 2024-03-25T18:22:00.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f78a4001a30 con 0x7f78b40b00b0 2024-03-25T18:22:00.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 monclient(hunting): dump: 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:00.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 client.?.objecter ms_handle_connect 0x7f78b406ab80 2024-03-25T18:22:00.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.706+0000 7f78aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:00.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2211975017 0 0) 0x7f78b41d4f20 con 0x7f78b40b00b0 2024-03-25T18:22:00.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 >> v1:172.21.15.87:6790/0 conn(0x7f78b406ab80 legacy=0x7f78b41388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:00.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 >> v1:172.21.15.87:6789/0 conn(0x7f78b4139a10 legacy=0x7f78b41d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 1 monclient: found mon.b 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b413efe0 con 0x7f78b40b00b0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 10 monclient: _renew_subs 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b413db80 con 0x7f78b40b00b0 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:00.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 10 monclient: _renew_subs 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b4139840 con 0x7f78b40b00b0 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.716+0000 7f78bc715640 1 librados: init done 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.717+0000 7f78bc715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.717+0000 7f78bc715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.717+0000 7f78bc715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:00.719 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.717+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b40627b0 con 0x7f78b40b00b0 2024-03-25T18:22:00.989 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78a4001d30 con 0x7f78b40b00b0 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f78a4003a60 con 0x7f78b40b00b0 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78aaffd640 10 monclient: dump: 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:00.990 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:00.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.988+0000 7f78b8cae640 4 set_mon_vals no callback set 2024-03-25T18:22:00.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.989+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f78a4003c40 con 0x7f78b40b00b0 2024-03-25T18:22:00.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.990+0000 7f78ba4b1640 10 monclient: get_auth_request con 0x7f789806dc10 auth_method 0 2024-03-25T18:22:00.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.990+0000 7f78aaffd640 10 client.4558.objecter ms_handle_connect 0x7f789806dc10 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f78a40549e0 con 0x7f78b40b00b0 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 10 client.4558.objecter ms_dispatch 0x7f78b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78a4050320 con 0x7f78b40b00b0 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:00.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:00.998+0000 7f78aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:01.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.248+0000 7f78bc715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:01.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.248+0000 7f78bc715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:01.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.248+0000 7f78bc715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:01.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.248+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78b4069f40 con 0x7f78b40b00b0 2024-03-25T18:22:01.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.269+0000 7f78aaffd640 1 -- 172.21.15.87:0/148600660 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f78a4052420 con 0x7f78b40b00b0 2024-03-25T18:22:01.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.269+0000 7f78aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:01.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.269+0000 7f78aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:01.271 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:01.271 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 10 librados: watch_flush enter 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 10 librados: watch_flush exit 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 10 client.4558.objecter successfully canceled tick 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 >> v1:172.21.15.171:6800/3417775508 conn(0x7f789806dc10 legacy=0x7f78980700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 10 monclient: shutdown 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 >> v1:172.21.15.171:6789/0 conn(0x7f78b40b00b0 legacy=0x7f78b41da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 shutdown_connections 2024-03-25T18:22:01.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 >> 172.21.15.87:0/148600660 conn(0x7f78b40b4a80 msgr2=0x7f78b40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:01.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.272+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 shutdown_connections 2024-03-25T18:22:01.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.273+0000 7f78bc715640 1 -- 172.21.15.87:0/148600660 wait complete. 2024-03-25T18:22:01.275 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:01.273+0000 7f78bc715640 1 librados: shutdown 2024-03-25T18:22:01.286 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:07.756 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:55.469 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.803 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:55.517 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.840 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:07:55.553 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.856 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:55.535 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.859 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:55.538 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.870 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:55.583 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.875 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:55.588 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.881 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:07:55.560 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:07.914 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:07:55.626 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:08.092 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:08.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.204+0000 7f54cf9dd640 10 monclient: get_monmap_and_config 2024-03-25T18:22:08.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.204+0000 7f54cf9dd640 10 monclient: build_initial_monmap 2024-03-25T18:22:08.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.204+0000 7f54cf9dd640 10 monclient: monmap: 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:08.205505+0000 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:08.205505+0000 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:08.206 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.205+0000 7f54cf9dd640 1 Processor -- start 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.205+0000 7f54cf9dd640 1 -- start start 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.205+0000 7f54cf9dd640 10 monclient: init 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-b con 0x7f54c80a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-c con 0x7f54c813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-a con 0x7f54c81430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c8142520 con 0x7f54c81430c0 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b28a0 con 0x7f54c813f710 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b2a30 con 0x7f54c80a98f0 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:08.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.206+0000 7f54cf9dd640 10 monclient(hunting): authenticate will time out at 8243.501020s 2024-03-25T18:22:08.210 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.209+0000 7f54cd779640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f54c80a98f0 0x7f54c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:47236/0 (socket says 172.21.15.87:47236) 2024-03-25T18:22:08.210 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.209+0000 7f54cd779640 1 -- 172.21.15.87:0/2303191624 learned_addr learned my addr 172.21.15.87:0/2303191624 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:08.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916799843 0 0) 0x7f54c80b2a30 con 0x7f54c80a98f0 2024-03-25T18:22:08.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4003520 con 0x7f54c80a98f0 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118203333 0 0) 0x7f54c80b28a0 con 0x7f54c813f710 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:22:08.225 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.223+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c80b2a30 con 0x7f54c813f710 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414004298 0 0) 0x7f54c8142520 con 0x7f54c81430c0 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c80b28a0 con 0x7f54c81430c0 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200770187 0 0) 0x7f54c80b2a30 con 0x7f54c813f710 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 >> v1:172.21.15.87:6790/0 conn(0x7f54c80a98f0 legacy=0x7f54c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 >> v1:172.21.15.87:6789/0 conn(0x7f54c81430c0 legacy=0x7f54c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:08.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 monclient: found mon.noname-c 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80b44c0 con 0x7f54c813f710 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f54b8001670 con 0x7f54c813f710 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.231+0000 7f54bffff640 10 monclient: dump: 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.234 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.232+0000 7f54cf9dd640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:22:08.235 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.232+0000 7f54cf9dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:08.239 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b80017d0 con 0x7f54c813f710 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 1 -- 172.21.15.87:0/2303191624 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f54b8001be0 con 0x7f54c813f710 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.238+0000 7f54bffff640 10 monclient: dump: 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:08.240 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:08.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 4 set_mon_vals no callback set 2024-03-25T18:22:08.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 10 monclient: shutdown 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 1 -- 172.21.15.87:0/2303191624 >> v1:172.21.15.171:6789/0 conn(0x7f54c813f710 legacy=0x7f54c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 1 -- 172.21.15.87:0/2303191624 shutdown_connections 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.239+0000 7f54cf9dd640 1 -- 172.21.15.87:0/2303191624 >> 172.21.15.87:0/2303191624 conn(0x7f54c80b7ea0 msgr2=0x7f54c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.240+0000 7f54cf9dd640 1 -- 172.21.15.87:0/2303191624 shutdown_connections 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.240+0000 7f54cf9dd640 1 -- 172.21.15.87:0/2303191624 wait complete. 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.240+0000 7f54cf9dd640 10 monclient: build_initial_monmap 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.240+0000 7f54cf9dd640 10 monclient: monmap: 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:08.243 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 librados: starting msgr at 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 librados: starting objecter 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 Processor -- start 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 -- start start 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 librados: setting wanted keys 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 librados: calling monclient init 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient: init 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-b con 0x7f54c80a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:08.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-a con 0x7f54c813f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient(hunting): picked mon.noname-c con 0x7f54c81430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c81390f0 con 0x7f54c813f710 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c8138dd0 con 0x7f54c80a98f0 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c81346b0 con 0x7f54c81430c0 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:08.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.241+0000 7f54cf9dd640 10 monclient(hunting): authenticate will time out at 8243.536583s 2024-03-25T18:22:08.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.261+0000 7f54cd779640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f54c80a98f0 0x7f54c8137e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52942/0 (socket says 172.21.15.87:52942) 2024-03-25T18:22:08.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.261+0000 7f54cd779640 1 -- 172.21.15.87:0/1279449047 learned_addr learned my addr 172.21.15.87:0/1279449047 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:08.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.262+0000 7f54bdffb640 10 client.?.objecter ms_handle_connect 0x7f54c80a98f0 2024-03-25T18:22:08.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.262+0000 7f54bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:08.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.268+0000 7f54bdffb640 10 client.?.objecter ms_handle_connect 0x7f54c813f710 2024-03-25T18:22:08.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.268+0000 7f54bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:08.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.268+0000 7f54bdffb640 10 client.?.objecter ms_handle_connect 0x7f54c81430c0 2024-03-25T18:22:08.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.268+0000 7f54bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089417640 0 0) 0x7f54c8138dd0 con 0x7f54c80a98f0 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:22:08.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.272+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac004eb0 con 0x7f54c80a98f0 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523368263 0 0) 0x7f54c81390f0 con 0x7f54c813f710 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:22:08.277 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.276+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c8138dd0 con 0x7f54c813f710 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402258570 0 0) 0x7f54c81346b0 con 0x7f54c81430c0 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:22:08.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.281+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c81390f0 con 0x7f54c81430c0 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112579885 0 0) 0x7f54ac004eb0 con 0x7f54c80a98f0 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c81346b0 con 0x7f54c80a98f0 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f54c4001950 con 0x7f54c80a98f0 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:08.287 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1994923602 0 0) 0x7f54c8138dd0 con 0x7f54c813f710 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54ac004eb0 con 0x7f54c813f710 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f54b8001ae0 con 0x7f54c813f710 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:08.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.286+0000 7f54bdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:08.289 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3632383465 0 0) 0x7f54ac004eb0 con 0x7f54c813f710 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 >> v1:172.21.15.87:6790/0 conn(0x7f54c81430c0 legacy=0x7f54c8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 >> v1:172.21.15.171:6789/0 conn(0x7f54c80a98f0 legacy=0x7f54c8137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 1 monclient: found mon.a 2024-03-25T18:22:08.298 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c8134840 con 0x7f54c813f710 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 10 monclient: _renew_subs 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8134a90 con 0x7f54c813f710 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 10 monclient: _renew_subs 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c81356f0 con 0x7f54c813f710 2024-03-25T18:22:08.299 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.297+0000 7f54cf9dd640 1 librados: init done 2024-03-25T18:22:08.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.298+0000 7f54cf9dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:08.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.298+0000 7f54cf9dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:08.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.298+0000 7f54cf9dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:08.300 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.298+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80627b0 con 0x7f54c813f710 2024-03-25T18:22:08.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.516+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b8001210 con 0x7f54c813f710 2024-03-25T18:22:08.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.516+0000 7f54bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:08.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.516+0000 7f54bf7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.592+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f54b8003a40 con 0x7f54c813f710 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.592+0000 7f54bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.592+0000 7f54bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.592+0000 7f54bdffb640 10 monclient: dump: 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:08.593 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:08.594 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:08.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.593+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f54b8019a10 con 0x7f54c813f710 2024-03-25T18:22:08.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.593+0000 7f54cd779640 10 monclient: get_auth_request con 0x7f54ac06dfb0 auth_method 0 2024-03-25T18:22:08.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.594+0000 7f54bdffb640 10 client.4560.objecter ms_handle_connect 0x7f54ac06dfb0 2024-03-25T18:22:08.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f54b8053710 con 0x7f54c813f710 2024-03-25T18:22:08.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 10 client.4560.objecter ms_dispatch 0x7f54c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:08.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:08.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:08.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-25T18:22:08.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54b804f050 con 0x7f54c813f710 2024-03-25T18:22:08.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:08.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.601+0000 7f54bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:08.834 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.833+0000 7f54cf9dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:08.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.833+0000 7f54cf9dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:08.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.833+0000 7f54cf9dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:08.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.833+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54c813de90 con 0x7f54c813f710 2024-03-25T18:22:08.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.919+0000 7f54bdffb640 1 -- 172.21.15.87:0/1279449047 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f54b8051150 con 0x7f54c813f710 2024-03-25T18:22:08.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.919+0000 7f54bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:08.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.919+0000 7f54bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:08.920 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:08.920 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 10 librados: watch_flush enter 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 10 librados: watch_flush exit 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 10 client.4560.objecter successfully canceled tick 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 >> v1:172.21.15.171:6800/3417775508 conn(0x7f54ac06dfb0 legacy=0x7f54ac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 10 monclient: shutdown 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 >> v1:172.21.15.87:6789/0 conn(0x7f54c813f710 legacy=0x7f54c81386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 shutdown_connections 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 >> 172.21.15.87:0/1279449047 conn(0x7f54c80b7ea0 msgr2=0x7f54c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:08.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 shutdown_connections 2024-03-25T18:22:08.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.921+0000 7f54cf9dd640 1 -- 172.21.15.87:0/1279449047 wait complete. 2024-03-25T18:22:08.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:08.922+0000 7f54cf9dd640 1 librados: shutdown 2024-03-25T18:22:08.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:15.740 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:15.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.854+0000 7f7b9de7b640 10 monclient: get_monmap_and_config 2024-03-25T18:22:15.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.854+0000 7f7b9de7b640 10 monclient: build_initial_monmap 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.854+0000 7f7b9de7b640 10 monclient: monmap: 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:15.855688+0000 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:15.855688+0000 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:15.856 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.855+0000 7f7b9de7b640 1 Processor -- start 2024-03-25T18:22:15.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.855+0000 7f7b9de7b640 1 -- start start 2024-03-25T18:22:15.857 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.855+0000 7f7b9de7b640 10 monclient: init 2024-03-25T18:22:15.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.856+0000 7f7b9de7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:15.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.856+0000 7f7b9de7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:15.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.856+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b980a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:15.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.856+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:15.858 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.856+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b981430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:15.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.857+0000 7f7b9de7b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98142520 con 0x7f7b980a98f0 2024-03-25T18:22:15.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.857+0000 7f7b9de7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980b28a0 con 0x7f7b981430c0 2024-03-25T18:22:15.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.857+0000 7f7b9de7b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980b2a30 con 0x7f7b9813f710 2024-03-25T18:22:15.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.857+0000 7f7b9de7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:15.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.857+0000 7f7b9de7b640 10 monclient(hunting): authenticate will time out at 8251.152007s 2024-03-25T18:22:15.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.860+0000 7f7b96ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f7b9813f710 0x7f7b981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:52086/0 (socket says 172.21.15.87:52086) 2024-03-25T18:22:15.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.860+0000 7f7b96ffd640 1 -- 172.21.15.87:0/742621275 learned_addr learned my addr 172.21.15.87:0/742621275 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456652092 0 0) 0x7f7b980b2a30 con 0x7f7b9813f710 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:22:15.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.867+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b80003520 con 0x7f7b9813f710 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210315410 0 0) 0x7f7b80003520 con 0x7f7b9813f710 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 >> v1:172.21.15.171:6789/0 conn(0x7f7b981430c0 legacy=0x7f7b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 >> v1:172.21.15.87:6789/0 conn(0x7f7b980a98f0 legacy=0x7f7b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 monclient: found mon.noname-b 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980b44c0 con 0x7f7b9813f710 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:15.881 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7b8c001690 con 0x7f7b9813f710 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b9de7b640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b9de7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b967fc640 10 monclient: dump: 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:15.882 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:15.883 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:15.883 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.879+0000 7f7b9de7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c001870 con 0x7f7b9813f710 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 1 -- 172.21.15.87:0/742621275 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7b8c001c80 con 0x7f7b9813f710 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b9de7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b9de7b640 4 set_mon_vals no callback set 2024-03-25T18:22:15.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.905+0000 7f7b967fc640 10 monclient: dump: 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:15.907 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 10 monclient: shutdown 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 1 -- 172.21.15.87:0/742621275 >> v1:172.21.15.87:6790/0 conn(0x7f7b9813f710 legacy=0x7f7b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 1 -- 172.21.15.87:0/742621275 shutdown_connections 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 1 -- 172.21.15.87:0/742621275 >> 172.21.15.87:0/742621275 conn(0x7f7b980b7ea0 msgr2=0x7f7b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 1 -- 172.21.15.87:0/742621275 shutdown_connections 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.906+0000 7f7b9de7b640 1 -- 172.21.15.87:0/742621275 wait complete. 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 10 monclient: build_initial_monmap 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 10 monclient: monmap: 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:15.909 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 librados: starting msgr at 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 librados: starting objecter 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 Processor -- start 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 -- start start 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 librados: setting wanted keys 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 1 librados: calling monclient init 2024-03-25T18:22:15.910 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.907+0000 7f7b9de7b640 10 monclient: init 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b980a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b981430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:15.911 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98139070 con 0x7f7b980a98f0 2024-03-25T18:22:15.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98133710 con 0x7f7b9813f710 2024-03-25T18:22:15.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b981338a0 con 0x7f7b981430c0 2024-03-25T18:22:15.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:15.912 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.908+0000 7f7b9de7b640 10 monclient(hunting): authenticate will time out at 8251.203602s 2024-03-25T18:22:15.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.914+0000 7f7b977fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f7b980a98f0 0x7f7b981378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:49992/0 (socket says 172.21.15.87:49992) 2024-03-25T18:22:15.915 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.914+0000 7f7b977fe640 1 -- 172.21.15.87:0/2596754738 learned_addr learned my addr 172.21.15.87:0/2596754738 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:15.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.915+0000 7f7b7ffff640 10 client.?.objecter ms_handle_connect 0x7f7b980a98f0 2024-03-25T18:22:15.916 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.915+0000 7f7b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:15.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.924+0000 7f7b7ffff640 10 client.?.objecter ms_handle_connect 0x7f7b9813f710 2024-03-25T18:22:15.924 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.924+0000 7f7b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817933671 0 0) 0x7f7b98139070 con 0x7f7b980a98f0 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:22:15.933 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.932+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84004eb0 con 0x7f7b980a98f0 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378895624 0 0) 0x7f7b98133710 con 0x7f7b9813f710 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:22:15.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.937+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98139070 con 0x7f7b9813f710 2024-03-25T18:22:15.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.944+0000 7f7b7ffff640 10 client.?.objecter ms_handle_connect 0x7f7b981430c0 2024-03-25T18:22:15.944 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.944+0000 7f7b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:15.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3538466239 0 0) 0x7f7b98139070 con 0x7f7b9813f710 2024-03-25T18:22:15.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b98133710 con 0x7f7b9813f710 2024-03-25T18:22:15.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7b8c001b80 con 0x7f7b9813f710 2024-03-25T18:22:15.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:15.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.959+0000 7f7b7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3792026578 0 0) 0x7f7b84004eb0 con 0x7f7b980a98f0 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b98139070 con 0x7f7b980a98f0 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7b880019d0 con 0x7f7b980a98f0 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.963+0000 7f7b7ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.964 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:15.965 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022825727 0 0) 0x7f7b981338a0 con 0x7f7b981430c0 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:22:15.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.967+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84004eb0 con 0x7f7b981430c0 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2404627095 0 0) 0x7f7b98133710 con 0x7f7b9813f710 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 >> v1:172.21.15.87:6790/0 conn(0x7f7b981430c0 legacy=0x7f7b98133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 >> v1:172.21.15.87:6789/0 conn(0x7f7b980a98f0 legacy=0x7f7b981378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 1 monclient: found mon.b 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b98133a30 con 0x7f7b9813f710 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:22:15.975 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 10 monclient: _renew_subs 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b980b7100 con 0x7f7b9813f710 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 10 monclient: _renew_subs 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98136e60 con 0x7f7b9813f710 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.973+0000 7f7b9de7b640 1 librados: init done 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.974+0000 7f7b9de7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:15.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.974+0000 7f7b9de7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:15.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.974+0000 7f7b9de7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:15.977 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:15.974+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b980627b0 con 0x7f7b9813f710 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c001240 con 0x7f7b9813f710 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7b8c004bf0 con 0x7f7b9813f710 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b7ffff640 10 monclient: dump: 2024-03-25T18:22:16.290 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:16.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.289+0000 7f7b95ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:16.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.290+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f7b8c01abc0 con 0x7f7b9813f710 2024-03-25T18:22:16.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.290+0000 7f7b977fe640 10 monclient: get_auth_request con 0x7f7b8406df40 auth_method 0 2024-03-25T18:22:16.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.291+0000 7f7b7ffff640 10 client.4570.objecter ms_handle_connect 0x7f7b8406df40 2024-03-25T18:22:16.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.294+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f7b8c0548a0 con 0x7f7b9813f710 2024-03-25T18:22:16.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.294+0000 7f7b7ffff640 10 client.4570.objecter ms_dispatch 0x7f7b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:16.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.294+0000 7f7b7ffff640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:16.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.294+0000 7f7b7ffff640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:16.295 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.294+0000 7f7b7ffff640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:22:16.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.296+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b8c0503b0 con 0x7f7b9813f710 2024-03-25T18:22:16.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.296+0000 7f7b7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:16.297 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.296+0000 7f7b7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:16.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.552+0000 7f7b9de7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:16.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.552+0000 7f7b9de7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:16.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.552+0000 7f7b9de7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:16.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.552+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b98074cc0 con 0x7f7b9813f710 2024-03-25T18:22:16.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.591+0000 7f7b7ffff640 1 -- 172.21.15.87:0/2596754738 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f7b8c0522e0 con 0x7f7b9813f710 2024-03-25T18:22:16.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.592+0000 7f7b7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:16.593 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.592+0000 7f7b7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:16.593 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:16.593 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:16.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 10 librados: watch_flush enter 2024-03-25T18:22:16.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 10 librados: watch_flush exit 2024-03-25T18:22:16.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 >> v1:172.21.15.171:6800/3417775508 conn(0x7f7b8406df40 legacy=0x7f7b84070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 10 monclient: shutdown 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 >> v1:172.21.15.171:6789/0 conn(0x7f7b9813f710 legacy=0x7f7b981daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 shutdown_connections 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 >> 172.21.15.87:0/2596754738 conn(0x7f7b980b7ea0 msgr2=0x7f7b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 shutdown_connections 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 -- 172.21.15.87:0/2596754738 wait complete. 2024-03-25T18:22:16.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:16.594+0000 7f7b9de7b640 1 librados: shutdown 2024-03-25T18:22:16.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:23.413 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:23.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.528+0000 7f635e1fe640 10 monclient: get_monmap_and_config 2024-03-25T18:22:23.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.528+0000 7f635e1fe640 10 monclient: build_initial_monmap 2024-03-25T18:22:23.530 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.528+0000 7f635e1fe640 10 monclient: monmap: 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:23.530366+0000 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:23.530366+0000 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:23.531 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.529+0000 7f635e1fe640 1 Processor -- start 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.529+0000 7f635e1fe640 1 -- start start 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.529+0000 7f635e1fe640 10 monclient: init 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-c con 0x7f63580a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-b con 0x7f635813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-a con 0x7f63581430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6358142520 con 0x7f63581430c0 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580b28a0 con 0x7f63580a98f0 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580b2a30 con 0x7f635813f710 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:23.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.530+0000 7f635e1fe640 10 monclient(hunting): authenticate will time out at 8258.826054s 2024-03-25T18:22:23.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.540+0000 7f6356ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f635813f710 0x7f63581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:56454/0 (socket says 172.21.15.87:56454) 2024-03-25T18:22:23.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.540+0000 7f6356ffd640 1 -- 172.21.15.87:0/3319479909 learned_addr learned my addr 172.21.15.87:0/3319479909 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:23.549 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026839384 0 0) 0x7f63580b2a30 con 0x7f635813f710 2024-03-25T18:22:23.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:22:23.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.547+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6340003520 con 0x7f635813f710 2024-03-25T18:22:23.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640979787 0 0) 0x7f6358142520 con 0x7f63581430c0 2024-03-25T18:22:23.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:22:23.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.555+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63580b2a30 con 0x7f63581430c0 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705517399 0 0) 0x7f63580b28a0 con 0x7f63580a98f0 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:22:23.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.557+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6358142520 con 0x7f63580a98f0 2024-03-25T18:22:23.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645080388 0 0) 0x7f63580b2a30 con 0x7f63581430c0 2024-03-25T18:22:23.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 >> v1:172.21.15.87:6790/0 conn(0x7f635813f710 legacy=0x7f63581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:23.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 >> v1:172.21.15.171:6789/0 conn(0x7f63580a98f0 legacy=0x7f63580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:23.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:23.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 monclient: found mon.noname-a 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63580b44c0 con 0x7f63581430c0 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f63500019f0 con 0x7f63581430c0 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f63567fc640 10 monclient: dump: 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.572 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f635e1fe640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:22:23.573 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.569+0000 7f635e1fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:23.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.581+0000 7f63567fc640 1 -- 172.21.15.87:0/3319479909 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6350001b50 con 0x7f63581430c0 2024-03-25T18:22:23.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.581+0000 7f63567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:23.583 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.581+0000 7f635e1fe640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:23.584 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.581+0000 7f635e1fe640 4 set_mon_vals no callback set 2024-03-25T18:22:23.584 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 10 monclient: shutdown 2024-03-25T18:22:23.584 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:23.584 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 1 -- 172.21.15.87:0/3319479909 >> v1:172.21.15.87:6789/0 conn(0x7f63581430c0 legacy=0x7f635814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:23.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 1 -- 172.21.15.87:0/3319479909 shutdown_connections 2024-03-25T18:22:23.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 1 -- 172.21.15.87:0/3319479909 >> 172.21.15.87:0/3319479909 conn(0x7f63580b7ea0 msgr2=0x7f63580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:23.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 1 -- 172.21.15.87:0/3319479909 shutdown_connections 2024-03-25T18:22:23.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.582+0000 7f635e1fe640 1 -- 172.21.15.87:0/3319479909 wait complete. 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.583+0000 7f635e1fe640 10 monclient: build_initial_monmap 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.583+0000 7f635e1fe640 10 monclient: monmap: 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:23.587 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.583+0000 7f635e1fe640 1 librados: starting msgr at 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.583+0000 7f635e1fe640 1 librados: starting objecter 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.583+0000 7f635e1fe640 1 Processor -- start 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 -- start start 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 librados: setting wanted keys 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 librados: calling monclient init 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient: init 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-a con 0x7f63580a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-b con 0x7f635813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient(hunting): picked mon.noname-c con 0x7f63581430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580abd30 con 0x7f63580a98f0 2024-03-25T18:22:23.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63581db8a0 con 0x7f635813f710 2024-03-25T18:22:23.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6358135220 con 0x7f63581430c0 2024-03-25T18:22:23.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:23.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.584+0000 7f635e1fe640 10 monclient(hunting): authenticate will time out at 8258.880510s 2024-03-25T18:22:23.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.600+0000 7f6356ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f635813f710 0x7f63581db190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:38098/0 (socket says 172.21.15.87:38098) 2024-03-25T18:22:23.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.600+0000 7f6356ffd640 1 -- 172.21.15.87:0/3444201612 learned_addr learned my addr 172.21.15.87:0/3444201612 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:23.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.605+0000 7f633ffff640 10 client.?.objecter ms_handle_connect 0x7f635813f710 2024-03-25T18:22:23.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.605+0000 7f633ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:23.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.611+0000 7f633ffff640 10 client.?.objecter ms_handle_connect 0x7f63580a98f0 2024-03-25T18:22:23.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.611+0000 7f633ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:23.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.611+0000 7f633ffff640 10 client.?.objecter ms_handle_connect 0x7f63581430c0 2024-03-25T18:22:23.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.611+0000 7f633ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:23.623 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756373263 0 0) 0x7f6358135220 con 0x7f63581430c0 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6344004eb0 con 0x7f63581430c0 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628943751 0 0) 0x7f63580abd30 con 0x7f63580a98f0 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:22:23.624 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.621+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6358135220 con 0x7f63580a98f0 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806097466 0 0) 0x7f63581db8a0 con 0x7f635813f710 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:22:23.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.625+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63580abd30 con 0x7f635813f710 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3637624673 0 0) 0x7f6358135220 con 0x7f63580a98f0 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63581db8a0 con 0x7f63580a98f0 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6348001870 con 0x7f63580a98f0 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.638 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.636+0000 7f633ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029427862 0 0) 0x7f63580abd30 con 0x7f635813f710 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6358135220 con 0x7f635813f710 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f634c001950 con 0x7f635813f710 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.642+0000 7f633ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.644 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:23.645 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:23.645 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:23.645 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384256622 0 0) 0x7f6344004eb0 con 0x7f63581430c0 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63580abd30 con 0x7f63581430c0 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6350001f20 con 0x7f63581430c0 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.648+0000 7f633ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:23.650 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.651 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:23.651 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:23.651 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:23.651 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3707133432 0 0) 0x7f63581db8a0 con 0x7f63580a98f0 2024-03-25T18:22:23.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 >> v1:172.21.15.87:6790/0 conn(0x7f63581430c0 legacy=0x7f6358133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:23.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 >> v1:172.21.15.171:6789/0 conn(0x7f635813f710 legacy=0x7f63581db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:23.653 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 1 monclient: found mon.a 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63581353b0 con 0x7f63580a98f0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f633ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 5 monclient: authenticate success, global_id 4569 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 10 monclient: _renew_subs 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6358134620 con 0x7f63580a98f0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 10 monclient: _renew_subs 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:23.654 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6358138ac0 con 0x7f63580a98f0 2024-03-25T18:22:23.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.651+0000 7f635e1fe640 1 librados: init done 2024-03-25T18:22:23.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.652+0000 7f635e1fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:23.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.652+0000 7f635e1fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:23.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.652+0000 7f635e1fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:23.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.652+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63581daa60 con 0x7f63580a98f0 2024-03-25T18:22:23.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.907+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6348003610 con 0x7f63580a98f0 2024-03-25T18:22:23.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.907+0000 7f633ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:23.909 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.907+0000 7f6355ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:23.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.968+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6348001d80 con 0x7f63580a98f0 2024-03-25T18:22:23.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.968+0000 7f633ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:23.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.968+0000 7f633ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:23.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.968+0000 7f633ffff640 10 monclient: dump: 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:23.971 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:23.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.969+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f6348019810 con 0x7f63580a98f0 2024-03-25T18:22:23.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.970+0000 7f6356ffd640 10 monclient: get_auth_request con 0x7f634406e0f0 auth_method 0 2024-03-25T18:22:23.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.970+0000 7f633ffff640 10 client.4569.objecter ms_handle_connect 0x7f634406e0f0 2024-03-25T18:22:23.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f63480535b0 con 0x7f63580a98f0 2024-03-25T18:22:23.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 10 client.4569.objecter ms_dispatch 0x7f6358000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f634804eef0 con 0x7f63580a98f0 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:23.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:23.976+0000 7f633ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:24.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.206+0000 7f635e1fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:24.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.206+0000 7f635e1fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:24.208 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.206+0000 7f635e1fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:22:24.209 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.206+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6358135540 con 0x7f63580a98f0 2024-03-25T18:22:24.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.256+0000 7f633ffff640 1 -- 172.21.15.87:0/3444201612 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f6348050ff0 con 0x7f63580a98f0 2024-03-25T18:22:24.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.256+0000 7f633ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:24.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.256+0000 7f633ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:24.259 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:24.259 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 10 librados: watch_flush enter 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 10 librados: watch_flush exit 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 10 client.4569.objecter successfully canceled tick 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 >> v1:172.21.15.171:6800/3417775508 conn(0x7f634406e0f0 legacy=0x7f63440705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 10 monclient: shutdown 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 >> v1:172.21.15.87:6789/0 conn(0x7f63580a98f0 legacy=0x7f6358137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 shutdown_connections 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 >> 172.21.15.87:0/3444201612 conn(0x7f63580b7ea0 msgr2=0x7f635813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:24.261 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 shutdown_connections 2024-03-25T18:22:24.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 -- 172.21.15.87:0/3444201612 wait complete. 2024-03-25T18:22:24.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:24.258+0000 7f635e1fe640 1 librados: shutdown 2024-03-25T18:22:24.273 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:31.078 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:31.194 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.193+0000 7fb769548640 10 monclient: get_monmap_and_config 2024-03-25T18:22:31.194 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.193+0000 7fb769548640 10 monclient: build_initial_monmap 2024-03-25T18:22:31.194 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.193+0000 7fb769548640 10 monclient: monmap: 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:31.194426+0000 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:31.194426+0000 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:31.195 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 1 Processor -- start 2024-03-25T18:22:31.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 1 -- start start 2024-03-25T18:22:31.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient: init 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient(hunting): picked mon.noname-b con 0x7fb764061830 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient(hunting): picked mon.noname-c con 0x7fb7641400e0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient(hunting): picked mon.noname-a con 0x7fb7641499d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764072600 con 0x7fb7641499d0 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764140c80 con 0x7fb7641400e0 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764072830 con 0x7fb764061830 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:31.197 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.194+0000 7fb769548640 10 monclient(hunting): authenticate will time out at 8266.489808s 2024-03-25T18:22:31.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.206+0000 7fb7627fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb7641400e0 0x7fb764140570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52026/0 (socket says 172.21.15.87:52026) 2024-03-25T18:22:31.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.206+0000 7fb7627fc640 1 -- 172.21.15.87:0/2767890465 learned_addr learned my addr 172.21.15.87:0/2767890465 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927045573 0 0) 0x7fb764140c80 con 0x7fb7641400e0 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:22:31.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.218+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb74c003520 con 0x7fb7641400e0 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082563712 0 0) 0x7fb764072600 con 0x7fb7641499d0 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:22:31.231 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.230+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764140c80 con 0x7fb7641499d0 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762206700 0 0) 0x7fb764072830 con 0x7fb764061830 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:22:31.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.240+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764072600 con 0x7fb764061830 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2491483474 0 0) 0x7fb764140c80 con 0x7fb7641499d0 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 >> v1:172.21.15.87:6790/0 conn(0x7fb764061830 legacy=0x7fb76413d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 >> v1:172.21.15.171:6789/0 conn(0x7fb7641400e0 legacy=0x7fb764140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 monclient: found mon.noname-a 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb764072b10 con 0x7fb7641499d0 2024-03-25T18:22:31.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb75c0019f0 con 0x7fb7641499d0 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb761ffb640 10 monclient: dump: 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:31.244 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:31.245 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:31.245 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb769548640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:22:31.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.242+0000 7fb769548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:31.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb75c001b50 con 0x7fb7641499d0 2024-03-25T18:22:31.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:31.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 1 -- 172.21.15.87:0/2767890465 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb75c001f60 con 0x7fb7641499d0 2024-03-25T18:22:31.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:31.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:31.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb761ffb640 10 monclient: dump: 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb769548640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:31.251 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.248+0000 7fb769548640 4 set_mon_vals no callback set 2024-03-25T18:22:31.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 10 monclient: shutdown 2024-03-25T18:22:31.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:31.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 1 -- 172.21.15.87:0/2767890465 >> v1:172.21.15.87:6789/0 conn(0x7fb7641499d0 legacy=0x7fb76414bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.253 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 1 -- 172.21.15.87:0/2767890465 shutdown_connections 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 1 -- 172.21.15.87:0/2767890465 >> 172.21.15.87:0/2767890465 conn(0x7fb7640b0ef0 msgr2=0x7fb7640b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 1 -- 172.21.15.87:0/2767890465 shutdown_connections 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.249+0000 7fb769548640 1 -- 172.21.15.87:0/2767890465 wait complete. 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.250+0000 7fb769548640 10 monclient: build_initial_monmap 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.250+0000 7fb769548640 10 monclient: monmap: 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.250+0000 7fb769548640 1 librados: starting msgr at 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.250+0000 7fb769548640 1 librados: starting objecter 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 1 Processor -- start 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 1 -- start start 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 1 librados: setting wanted keys 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 1 librados: calling monclient init 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 10 monclient: init 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 10 monclient(hunting): picked mon.noname-b con 0x7fb764061830 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.251+0000 7fb769548640 10 monclient(hunting): picked mon.noname-c con 0x7fb7641400e0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 10 monclient(hunting): picked mon.noname-a con 0x7fb7641499d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7641369a0 con 0x7fb7641499d0 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7641dab80 con 0x7fb764061830 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76406f640 con 0x7fb7641400e0 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:31.255 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.252+0000 7fb769548640 10 monclient(hunting): authenticate will time out at 8266.546915s 2024-03-25T18:22:31.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.256+0000 7fb762ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb764061830 0x7fb7641db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52066/0 (socket says 172.21.15.87:52066) 2024-03-25T18:22:31.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.256+0000 7fb762ffd640 1 -- 172.21.15.87:0/2877777075 learned_addr learned my addr 172.21.15.87:0/2877777075 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:31.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.257+0000 7fb74b7fe640 10 client.?.objecter ms_handle_connect 0x7fb764061830 2024-03-25T18:22:31.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.257+0000 7fb74b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409776914 0 0) 0x7fb7641dab80 con 0x7fb764061830 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:22:31.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.262+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750004eb0 con 0x7fb764061830 2024-03-25T18:22:31.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.266+0000 7fb74b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7641499d0 2024-03-25T18:22:31.267 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.266+0000 7fb74b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503425 0 0) 0x7fb7641369a0 con 0x7fb7641499d0 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:22:31.276 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.275+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7641dab80 con 0x7fb7641499d0 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909225106 0 0) 0x7fb750004eb0 con 0x7fb764061830 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7641369a0 con 0x7fb764061830 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb754001a10 con 0x7fb764061830 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.278 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.277+0000 7fb74b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:31.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.284+0000 7fb74b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7641400e0 2024-03-25T18:22:31.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.284+0000 7fb74b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760546294 0 0) 0x7fb7641369a0 con 0x7fb764061830 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 >> v1:172.21.15.87:6790/0 conn(0x7fb7641400e0 legacy=0x7fb764133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 >> v1:172.21.15.87:6789/0 conn(0x7fb7641499d0 legacy=0x7fb764134fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 1 monclient: found mon.b 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76406f7d0 con 0x7fb764061830 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:31.291 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb74b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 10 monclient: _renew_subs 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7641db7b0 con 0x7fb764061830 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 10 monclient: _renew_subs 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb764133e80 con 0x7fb764061830 2024-03-25T18:22:31.292 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.290+0000 7fb769548640 1 librados: init done 2024-03-25T18:22:31.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.291+0000 7fb769548640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:31.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.291+0000 7fb769548640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:31.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.291+0000 7fb769548640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:31.293 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.291+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7641dad10 con 0x7fb764061830 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb754001c70 con 0x7fb764061830 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fb754003a60 con 0x7fb764061830 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb74b7fe640 10 monclient: dump: 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:31.587 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.586+0000 7fb7617fa640 4 set_mon_vals no callback set 2024-03-25T18:22:31.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.587+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fb754003c80 con 0x7fb764061830 2024-03-25T18:22:31.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.587+0000 7fb7627fc640 10 monclient: get_auth_request con 0x7fb75006dd10 auth_method 0 2024-03-25T18:22:31.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.587+0000 7fb74b7fe640 10 client.4582.objecter ms_handle_connect 0x7fb75006dd10 2024-03-25T18:22:31.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.593+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fb754054850 con 0x7fb764061830 2024-03-25T18:22:31.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.593+0000 7fb74b7fe640 10 client.4582.objecter ms_dispatch 0x7fb764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:31.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.593+0000 7fb74b7fe640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:31.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.593+0000 7fb74b7fe640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:31.594 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.593+0000 7fb74b7fe640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:22:31.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.594+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb754050360 con 0x7fb764061830 2024-03-25T18:22:31.595 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.594+0000 7fb74b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:31.596 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.594+0000 7fb74b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:31.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.803+0000 7fb769548640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:31.804 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.803+0000 7fb769548640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:31.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.803+0000 7fb769548640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:31.805 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.803+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7640627b0 con 0x7fb764061830 2024-03-25T18:22:31.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.847+0000 7fb74b7fe640 1 -- 172.21.15.87:0/2877777075 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fb754052290 con 0x7fb764061830 2024-03-25T18:22:31.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.847+0000 7fb74b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:31.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.847+0000 7fb74b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:31.849 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:31.849 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:31.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.849+0000 7fb769548640 10 librados: watch_flush enter 2024-03-25T18:22:31.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.849+0000 7fb769548640 10 librados: watch_flush exit 2024-03-25T18:22:31.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.849+0000 7fb769548640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:22:31.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.849+0000 7fb769548640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 >> v1:172.21.15.171:6800/3417775508 conn(0x7fb75006dd10 legacy=0x7fb7500701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 10 monclient: shutdown 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 >> v1:172.21.15.171:6789/0 conn(0x7fb764061830 legacy=0x7fb7641db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 shutdown_connections 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 >> 172.21.15.87:0/2877777075 conn(0x7fb7640b0ef0 msgr2=0x7fb7640b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 shutdown_connections 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 -- 172.21.15.87:0/2877777075 wait complete. 2024-03-25T18:22:31.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:31.850+0000 7fb769548640 1 librados: shutdown 2024-03-25T18:22:31.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:32.811 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:20.524 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.816 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:20.529 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.852 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:20.566 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.871 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:20.550 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.874 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:20.553 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.895 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:20.608 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.899 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:20.578 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.910 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:20.623 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.934 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:20.647 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.672 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:38.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.789+0000 7f725e0ff640 10 monclient: get_monmap_and_config 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.789+0000 7f725e0ff640 10 monclient: build_initial_monmap 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.789+0000 7f725e0ff640 10 monclient: monmap: 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:38.790331+0000 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:38.790331+0000 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:38.791 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.790+0000 7f725e0ff640 1 Processor -- start 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.790+0000 7f725e0ff640 1 -- start start 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.790+0000 7f725e0ff640 10 monclient: init 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f72580a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f725813f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f72581430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7258142520 con 0x7f725813f710 2024-03-25T18:22:38.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580b28a0 con 0x7f72580a98f0 2024-03-25T18:22:38.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580b2a30 con 0x7f72581430c0 2024-03-25T18:22:38.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:38.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.791+0000 7f725e0ff640 10 monclient(hunting): authenticate will time out at 8274.086113s 2024-03-25T18:22:38.799 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.798+0000 7f72577fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72580a98f0 0x7f72580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:37148/0 (socket says 172.21.15.87:37148) 2024-03-25T18:22:38.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.799+0000 7f72577fe640 1 -- 172.21.15.87:0/884830851 learned_addr learned my addr 172.21.15.87:0/884830851 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:38.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277183390 0 0) 0x7f72580b28a0 con 0x7f72580a98f0 2024-03-25T18:22:38.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:38.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:38.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:38.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:22:38.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.810+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7240003520 con 0x7f72580a98f0 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731127138 0 0) 0x7f72580b2a30 con 0x7f72581430c0 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:22:38.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.827+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72580b28a0 con 0x7f72581430c0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573174898 0 0) 0x7f72580b28a0 con 0x7f72581430c0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 >> v1:172.21.15.171:6789/0 conn(0x7f72580a98f0 legacy=0x7f72580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 >> v1:172.21.15.87:6789/0 conn(0x7f725813f710 legacy=0x7f72581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 monclient: found mon.noname-b 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580b44c0 con 0x7f72581430c0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:38.835 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f72500019f0 con 0x7f72581430c0 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f72567fc640 10 monclient: dump: 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:38.836 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f725e0ff640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:22:38.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.833+0000 7f725e0ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:38.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7250001b50 con 0x7f72581430c0 2024-03-25T18:22:38.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:38.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 1 -- 172.21.15.87:0/884830851 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7250001f20 con 0x7f72581430c0 2024-03-25T18:22:38.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f725e0ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:38.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f725e0ff640 4 set_mon_vals no callback set 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.847+0000 7f72567fc640 10 monclient: dump: 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:38.849 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 10 monclient: shutdown 2024-03-25T18:22:38.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 1 -- 172.21.15.87:0/884830851 >> v1:172.21.15.87:6790/0 conn(0x7f72581430c0 legacy=0x7f725814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 1 -- 172.21.15.87:0/884830851 shutdown_connections 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 1 -- 172.21.15.87:0/884830851 >> 172.21.15.87:0/884830851 conn(0x7f72580b7ea0 msgr2=0x7f72580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 1 -- 172.21.15.87:0/884830851 shutdown_connections 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 1 -- 172.21.15.87:0/884830851 wait complete. 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 10 monclient: build_initial_monmap 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.848+0000 7f725e0ff640 10 monclient: monmap: 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:38.852 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 librados: starting msgr at 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 librados: starting objecter 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 Processor -- start 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 -- start start 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 librados: setting wanted keys 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 librados: calling monclient init 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient: init 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:38.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-a con 0x7f72580a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-c con 0x7f725813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient(hunting): picked mon.noname-b con 0x7f72581430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580ae940 con 0x7f72580a98f0 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580b0870 con 0x7f72581430c0 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580b0b50 con 0x7f725813f710 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:38.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.849+0000 7f725e0ff640 10 monclient(hunting): authenticate will time out at 8274.144806s 2024-03-25T18:22:38.862 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.861+0000 7f72577fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f72580a98f0 0x7f72580af0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:48854/0 (socket says 172.21.15.87:48854) 2024-03-25T18:22:38.862 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.861+0000 7f72577fe640 1 -- 172.21.15.87:0/1459184924 learned_addr learned my addr 172.21.15.87:0/1459184924 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:38.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.862+0000 7f723ffff640 10 client.?.objecter ms_handle_connect 0x7f72580a98f0 2024-03-25T18:22:38.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.862+0000 7f723ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:38.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.868+0000 7f723ffff640 10 client.?.objecter ms_handle_connect 0x7f725813f710 2024-03-25T18:22:38.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.868+0000 7f723ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:38.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 client.?.objecter ms_handle_connect 0x7f72581430c0 2024-03-25T18:22:38.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:38.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534373651 0 0) 0x7f72580ae940 con 0x7f72580a98f0 2024-03-25T18:22:38.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:38.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:38.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:38.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:22:38.875 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.870+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7244004eb0 con 0x7f72580a98f0 2024-03-25T18:22:38.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723398136 0 0) 0x7f72580b0b50 con 0x7f725813f710 2024-03-25T18:22:38.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:38.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:38.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:38.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:22:38.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.877+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72580ae940 con 0x7f725813f710 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954251028 0 0) 0x7f72580b0870 con 0x7f72581430c0 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:22:38.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.887+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72580b0b50 con 0x7f72581430c0 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3264084599 0 0) 0x7f72580ae940 con 0x7f725813f710 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72580b0870 con 0x7f725813f710 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f724c0018b0 con 0x7f725813f710 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:38.892 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.893 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.891+0000 7f723ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:38.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3182720354 0 0) 0x7f7244004eb0 con 0x7f72580a98f0 2024-03-25T18:22:38.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72580ae940 con 0x7f72580a98f0 2024-03-25T18:22:38.897 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f7248001950 con 0x7f72580a98f0 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.896+0000 7f723ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:38.898 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1821313925 0 0) 0x7f72580b0870 con 0x7f725813f710 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 >> v1:172.21.15.171:6789/0 conn(0x7f72581430c0 legacy=0x7f72581d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 >> v1:172.21.15.87:6789/0 conn(0x7f72580a98f0 legacy=0x7f72580af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 1 monclient: found mon.c 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:38.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580aeb70 con 0x7f725813f710 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f723ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f725e0ff640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f725e0ff640 10 monclient: _renew_subs 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f725e0ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.903+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72581d82b0 con 0x7f725813f710 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.904+0000 7f725e0ff640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.904+0000 7f725e0ff640 10 monclient: _renew_subs 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.904+0000 7f725e0ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.904+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72580af9d0 con 0x7f725813f710 2024-03-25T18:22:38.906 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.904+0000 7f725e0ff640 1 librados: init done 2024-03-25T18:22:38.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.905+0000 7f725e0ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:38.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.905+0000 7f725e0ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:38.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.905+0000 7f725e0ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:38.907 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:38.905+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725814f2d0 con 0x7f725813f710 2024-03-25T18:22:39.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f724c003610 con 0x7f725813f710 2024-03-25T18:22:39.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f724c0039e0 con 0x7f725813f710 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f723ffff640 10 monclient: dump: 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:39.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.218+0000 7f7255ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:39.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.219+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f724c003c00 con 0x7f725813f710 2024-03-25T18:22:39.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.220+0000 7f72577fe640 10 monclient: get_auth_request con 0x7f724406df70 auth_method 0 2024-03-25T18:22:39.221 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.220+0000 7f723ffff640 10 client.4592.objecter ms_handle_connect 0x7f724406df70 2024-03-25T18:22:39.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f724c053540 con 0x7f725813f710 2024-03-25T18:22:39.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 10 client.4592.objecter ms_dispatch 0x7f7258000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:39.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:39.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:39.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T18:22:39.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f724c04f050 con 0x7f725813f710 2024-03-25T18:22:39.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:39.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.225+0000 7f723ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:39.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.456+0000 7f725e0ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:39.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.456+0000 7f725e0ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:39.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.456+0000 7f725e0ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:39.457 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.456+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72580aed00 con 0x7f725813f710 2024-03-25T18:22:39.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.493+0000 7f723ffff640 1 -- 172.21.15.87:0/1459184924 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f724c056ba0 con 0x7f725813f710 2024-03-25T18:22:39.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.493+0000 7f723ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:39.494 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.493+0000 7f723ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:39.494 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:39.494 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:39.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 10 librados: watch_flush enter 2024-03-25T18:22:39.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 10 librados: watch_flush exit 2024-03-25T18:22:39.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T18:22:39.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 10 client.4592.objecter successfully canceled tick 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 >> v1:172.21.15.171:6800/3417775508 conn(0x7f724406df70 legacy=0x7f7244070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 10 monclient: shutdown 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 >> v1:172.21.15.87:6790/0 conn(0x7f725813f710 legacy=0x7f72581d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 shutdown_connections 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.496+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 >> 172.21.15.87:0/1459184924 conn(0x7f72580b7ea0 msgr2=0x7f7258074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.497+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 shutdown_connections 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.497+0000 7f725e0ff640 1 -- 172.21.15.87:0/1459184924 wait complete. 2024-03-25T18:22:39.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:39.497+0000 7f725e0ff640 1 librados: shutdown 2024-03-25T18:22:39.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:46.315 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.428+0000 7efc75510640 10 monclient: get_monmap_and_config 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.428+0000 7efc75510640 10 monclient: build_initial_monmap 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.429+0000 7efc75510640 10 monclient: monmap: 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:46.429856+0000 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:46.429856+0000 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:46.430 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:46.431 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:46.431 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:46.431 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:46.431 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.429+0000 7efc75510640 1 Processor -- start 2024-03-25T18:22:46.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.429+0000 7efc75510640 1 -- start start 2024-03-25T18:22:46.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.429+0000 7efc75510640 10 monclient: init 2024-03-25T18:22:46.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient(hunting): picked mon.noname-a con 0x7efc700a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient(hunting): picked mon.noname-b con 0x7efc7013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient(hunting): picked mon.noname-c con 0x7efc701430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc70142520 con 0x7efc700a98f0 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc700b28a0 con 0x7efc701430c0 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc700b2a30 con 0x7efc7013f710 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:46.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.430+0000 7efc75510640 10 monclient(hunting): authenticate will time out at 8281.725517s 2024-03-25T18:22:46.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.442+0000 7efc6effd640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7efc700a98f0 0x7efc700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53360/0 (socket says 172.21.15.87:53360) 2024-03-25T18:22:46.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.442+0000 7efc6effd640 1 -- 172.21.15.87:0/1520013983 learned_addr learned my addr 172.21.15.87:0/1520013983 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:46.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039900703 0 0) 0x7efc70142520 con 0x7efc700a98f0 2024-03-25T18:22:46.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:22:46.462 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.460+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5c003520 con 0x7efc700a98f0 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325479201 0 0) 0x7efc700b28a0 con 0x7efc701430c0 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:22:46.466 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.465+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc70142520 con 0x7efc701430c0 2024-03-25T18:22:46.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156081917 0 0) 0x7efc700b2a30 con 0x7efc7013f710 2024-03-25T18:22:46.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:22:46.475 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.473+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc700b28a0 con 0x7efc7013f710 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180641125 0 0) 0x7efc70142520 con 0x7efc701430c0 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 >> v1:172.21.15.87:6790/0 conn(0x7efc7013f710 legacy=0x7efc701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 >> v1:172.21.15.87:6789/0 conn(0x7efc700a98f0 legacy=0x7efc700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700b44c0 con 0x7efc701430c0 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:46.479 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7efc680019f0 con 0x7efc701430c0 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc6e7fc640 10 monclient: dump: 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc75510640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:22:46.480 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.477+0000 7efc75510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc68001bd0 con 0x7efc701430c0 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 1 -- 172.21.15.87:0/1520013983 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7efc68001fe0 con 0x7efc701430c0 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc75510640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc75510640 4 set_mon_vals no callback set 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:46.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.510+0000 7efc6e7fc640 10 monclient: dump: 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:46.512 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 10 monclient: shutdown 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 1 -- 172.21.15.87:0/1520013983 >> v1:172.21.15.171:6789/0 conn(0x7efc701430c0 legacy=0x7efc7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 1 -- 172.21.15.87:0/1520013983 shutdown_connections 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 1 -- 172.21.15.87:0/1520013983 >> 172.21.15.87:0/1520013983 conn(0x7efc700b7ea0 msgr2=0x7efc700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:46.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 1 -- 172.21.15.87:0/1520013983 shutdown_connections 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.511+0000 7efc75510640 1 -- 172.21.15.87:0/1520013983 wait complete. 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 10 monclient: build_initial_monmap 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 10 monclient: monmap: 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:46.514 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 librados: starting msgr at 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 librados: starting objecter 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 Processor -- start 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 -- start start 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 librados: setting wanted keys 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 1 librados: calling monclient init 2024-03-25T18:22:46.515 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.512+0000 7efc75510640 10 monclient: init 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient(hunting): picked mon.noname-a con 0x7efc700a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient(hunting): picked mon.noname-c con 0x7efc7013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient(hunting): picked mon.noname-b con 0x7efc701430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc700ad460 con 0x7efc700a98f0 2024-03-25T18:22:46.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc700b1670 con 0x7efc701430c0 2024-03-25T18:22:46.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc701d7a90 con 0x7efc7013f710 2024-03-25T18:22:46.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:46.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.513+0000 7efc75510640 10 monclient(hunting): authenticate will time out at 8281.808276s 2024-03-25T18:22:46.519 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.519+0000 7efc67fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7efc7013f710 0x7efc701d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:50662/0 (socket says 172.21.15.87:50662) 2024-03-25T18:22:46.520 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.519+0000 7efc67fff640 1 -- 172.21.15.87:0/3837974730 learned_addr learned my addr 172.21.15.87:0/3837974730 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:46.520 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.520+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc7013f710 2024-03-25T18:22:46.520 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.520+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582546945 0 0) 0x7efc701d7a90 con 0x7efc7013f710 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:22:46.526 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.525+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50004eb0 con 0x7efc7013f710 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4195146199 0 0) 0x7efc50004eb0 con 0x7efc7013f710 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc701d7a90 con 0x7efc7013f710 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7efc58001a10 con 0x7efc7013f710 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 10 monclient(hunting): dump: 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:46.537 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.538 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.536+0000 7efc677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:46.539 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.537+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc700a98f0 2024-03-25T18:22:46.539 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.537+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:46.539 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.537+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc701430c0 2024-03-25T18:22:46.539 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.537+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:46.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766848628 0 0) 0x7efc700b1670 con 0x7efc701430c0 2024-03-25T18:22:46.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:22:46.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.541+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50004eb0 con 0x7efc701430c0 2024-03-25T18:22:46.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276704530 0 0) 0x7efc5c003520 con 0x7efc700a98f0 2024-03-25T18:22:46.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:46.545 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:46.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:46.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:22:46.546 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.544+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc700b1670 con 0x7efc700a98f0 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2919097592 0 0) 0x7efc701d7a90 con 0x7efc7013f710 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 >> v1:172.21.15.171:6789/0 conn(0x7efc701430c0 legacy=0x7efc7014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 >> v1:172.21.15.87:6789/0 conn(0x7efc700a98f0 legacy=0x7efc701d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 1 monclient: found mon.c 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7014ece0 con 0x7efc7013f710 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:46.552 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc75510640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc75510640 10 monclient: _renew_subs 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc75510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.550+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7014dea0 con 0x7efc7013f710 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 monclient: _renew_subs 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc700b13e0 con 0x7efc7013f710 2024-03-25T18:22:46.553 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 1 librados: init done 2024-03-25T18:22:46.554 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:46.554 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:46.554 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:46.554 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.551+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc700627b0 con 0x7efc7013f710 2024-03-25T18:22:46.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc58001d30 con 0x7efc7013f710 2024-03-25T18:22:46.558 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7efc58003a60 con 0x7efc7013f710 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc677fe640 10 monclient: dump: 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:46.559 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:46.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.557+0000 7efc6dffb640 4 set_mon_vals no callback set 2024-03-25T18:22:46.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.844+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7efc58003bc0 con 0x7efc7013f710 2024-03-25T18:22:46.846 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.845+0000 7efc6effd640 10 monclient: get_auth_request con 0x7efc5006de20 auth_method 0 2024-03-25T18:22:46.846 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.845+0000 7efc677fe640 10 client.4598.objecter ms_handle_connect 0x7efc5006de20 2024-03-25T18:22:46.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7efc580548a0 con 0x7efc7013f710 2024-03-25T18:22:46.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 10 client.4598.objecter ms_dispatch 0x7efc70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc580501e0 con 0x7efc7013f710 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:46.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:46.854+0000 7efc677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:47.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.085+0000 7efc75510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:47.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.085+0000 7efc75510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:47.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.085+0000 7efc75510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:22:47.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.085+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc7013b430 con 0x7efc7013f710 2024-03-25T18:22:47.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.185+0000 7efc677fe640 1 -- 172.21.15.87:0/3837974730 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7efc580522e0 con 0x7efc7013f710 2024-03-25T18:22:47.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.185+0000 7efc677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:47.187 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.185+0000 7efc677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:47.187 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:47.187 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:47.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 10 librados: watch_flush enter 2024-03-25T18:22:47.189 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 10 librados: watch_flush exit 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 10 client.4598.objecter successfully canceled tick 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 >> v1:172.21.15.171:6800/3417775508 conn(0x7efc5006de20 legacy=0x7efc500702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 10 monclient: shutdown 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 >> v1:172.21.15.87:6790/0 conn(0x7efc7013f710 legacy=0x7efc701d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 shutdown_connections 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 >> 172.21.15.87:0/3837974730 conn(0x7efc700b7ea0 msgr2=0x7efc70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 shutdown_connections 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 -- 172.21.15.87:0/3837974730 wait complete. 2024-03-25T18:22:47.190 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:47.188+0000 7efc75510640 1 librados: shutdown 2024-03-25T18:22:47.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:52.889 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:40.602 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.901 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:40.615 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.932 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:08:40.645 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.957 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:40.636 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.963 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:40.676 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.967 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:40.646 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.972 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:40.685 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.994 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:08:40.672 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.004 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:08:40.717 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.008 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:54.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.127+0000 7f110f6d6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:54.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.127+0000 7f110f6d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.127+0000 7f110f6d6640 10 monclient: monmap: 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:22:54.128597+0000 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:22:54.128597+0000 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:22:54.129 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:22:54.130 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.128+0000 7f110f6d6640 1 Processor -- start 2024-03-25T18:22:54.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.128+0000 7f110f6d6640 1 -- start start 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.128+0000 7f110f6d6640 10 monclient: init 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f11080a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f110813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f11081430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1108142520 con 0x7f11080a98f0 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11080b28a0 con 0x7f110813f710 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11080b2a30 con 0x7f11081430c0 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:54.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.129+0000 7f110f6d6640 10 monclient(hunting): authenticate will time out at 8289.424575s 2024-03-25T18:22:54.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.145+0000 7f110dc73640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f11081430c0 0x7f110814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:43884/0 (socket says 172.21.15.87:43884) 2024-03-25T18:22:54.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.145+0000 7f110dc73640 1 -- 172.21.15.87:0/3564727326 learned_addr learned my addr 172.21.15.87:0/3564727326 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:54.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576332849 0 0) 0x7f11080b2a30 con 0x7f11081430c0 2024-03-25T18:22:54.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:22:54.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.156+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003520 con 0x7f11081430c0 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.165+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936033437 0 0) 0x7f11080b28a0 con 0x7f110813f710 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.165+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.165+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.165+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.165+0000 7f10fffff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:22:54.167 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.166+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11080b2a30 con 0x7f110813f710 2024-03-25T18:22:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612014581 0 0) 0x7f1108142520 con 0x7f11080a98f0 2024-03-25T18:22:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:22:54.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.175+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11080b28a0 con 0x7f11080a98f0 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852389507 0 0) 0x7f11080b2a30 con 0x7f110813f710 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 >> v1:172.21.15.87:6790/0 conn(0x7f11081430c0 legacy=0x7f110814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 >> v1:172.21.15.87:6789/0 conn(0x7f11080a98f0 legacy=0x7f11080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 monclient: found mon.noname-c 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080b44c0 con 0x7f110813f710 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:54.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f10f8001670 con 0x7f110813f710 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f10fffff640 10 monclient: dump: 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.182 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.183 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f110f6d6640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:22:54.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.179+0000 7f110f6d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:54.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f80017d0 con 0x7f110813f710 2024-03-25T18:22:54.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 1 -- 172.21.15.87:0/3564727326 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f10f8001be0 con 0x7f110813f710 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f10fffff640 10 monclient: dump: 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.202 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.203 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.203 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f110f6d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:54.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.200+0000 7f110f6d6640 4 set_mon_vals no callback set 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 10 monclient: shutdown 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 1 -- 172.21.15.87:0/3564727326 >> v1:172.21.15.171:6789/0 conn(0x7f110813f710 legacy=0x7f11081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 1 -- 172.21.15.87:0/3564727326 shutdown_connections 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 1 -- 172.21.15.87:0/3564727326 >> 172.21.15.87:0/3564727326 conn(0x7f11080b7ea0 msgr2=0x7f11080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 1 -- 172.21.15.87:0/3564727326 shutdown_connections 2024-03-25T18:22:54.204 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.201+0000 7f110f6d6640 1 -- 172.21.15.87:0/3564727326 wait complete. 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.202+0000 7f110f6d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.202+0000 7f110f6d6640 10 monclient: monmap: 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:22:54.205 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 librados: starting msgr at 2024-03-25T18:22:54.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 librados: starting objecter 2024-03-25T18:22:54.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 Processor -- start 2024-03-25T18:22:54.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 -- start start 2024-03-25T18:22:54.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 librados: setting wanted keys 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 1 librados: calling monclient init 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.203+0000 7f110f6d6640 10 monclient: init 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f11080a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f110813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f11081430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11081db640 con 0x7f11081430c0 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11081dc4c0 con 0x7f11080a98f0 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110814eb60 con 0x7f110813f710 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:54.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.204+0000 7f110f6d6640 10 monclient(hunting): authenticate will time out at 8289.499203s 2024-03-25T18:22:54.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.217+0000 7f110d472640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f11080a98f0 0x7f11081dbdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:42838/0 (socket says 172.21.15.87:42838) 2024-03-25T18:22:54.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.217+0000 7f110d472640 1 -- 172.21.15.87:0/2504046072 learned_addr learned my addr 172.21.15.87:0/2504046072 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:22:54.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.217+0000 7f10fdffb640 10 client.?.objecter ms_handle_connect 0x7f11081430c0 2024-03-25T18:22:54.218 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.217+0000 7f10fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:54.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.219+0000 7f10fdffb640 10 client.?.objecter ms_handle_connect 0x7f11080a98f0 2024-03-25T18:22:54.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.219+0000 7f10fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:54.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.219+0000 7f10fdffb640 10 client.?.objecter ms_handle_connect 0x7f110813f710 2024-03-25T18:22:54.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.219+0000 7f10fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365352033 0 0) 0x7f11081dc4c0 con 0x7f11080a98f0 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:22:54.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.231+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec004eb0 con 0x7f11080a98f0 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19233946 0 0) 0x7f110814eb60 con 0x7f110813f710 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:22:54.234 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.232+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11081dc4c0 con 0x7f110813f710 2024-03-25T18:22:54.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436952665 0 0) 0x7f11081db640 con 0x7f11081430c0 2024-03-25T18:22:54.242 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:54.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:54.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:54.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:22:54.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.241+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110814eb60 con 0x7f11081430c0 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696641355 0 0) 0x7f10ec004eb0 con 0x7f11080a98f0 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11081db640 con 0x7f11080a98f0 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1104001a10 con 0x7f11080a98f0 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:54.259 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.260 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.258+0000 7f10fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274813673 0 0) 0x7f11081dc4c0 con 0x7f110813f710 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec004eb0 con 0x7f110813f710 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f10f8001ae0 con 0x7f110813f710 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.263+0000 7f10fdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:54.264 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.265 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261715942 0 0) 0x7f110814eb60 con 0x7f11081430c0 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11081dc4c0 con 0x7f11081430c0 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1100001a30 con 0x7f11081430c0 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.273+0000 7f10fdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.274 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.275 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4177133875 0 0) 0x7f11081db640 con 0x7f11080a98f0 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 >> v1:172.21.15.87:6790/0 conn(0x7f110813f710 legacy=0x7f11080716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 >> v1:172.21.15.87:6789/0 conn(0x7f11081430c0 legacy=0x7f11081e6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 1 monclient: found mon.b 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110814e710 con 0x7f11080a98f0 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f10fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 10 monclient: _renew_subs 2024-03-25T18:22:54.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110806eb30 con 0x7f11080a98f0 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 10 monclient: _renew_subs 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1108071fd0 con 0x7f11080a98f0 2024-03-25T18:22:54.286 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.283+0000 7f110f6d6640 1 librados: init done 2024-03-25T18:22:54.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.284+0000 7f110f6d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:54.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.284+0000 7f110f6d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:54.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.284+0000 7f110f6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.284+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110806fc30 con 0x7f11080a98f0 2024-03-25T18:22:54.597 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1104001c70 con 0x7f11080a98f0 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f11040038c0 con 0x7f11080a98f0 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10fdffb640 10 monclient: dump: 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:54.598 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:22:54.599 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:22:54.599 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:22:54.599 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:22:54.599 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:22:54.599 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.596+0000 7f10ff7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:54.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.597+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f1104003ae0 con 0x7f11080a98f0 2024-03-25T18:22:54.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.598+0000 7f110cc71640 10 monclient: get_auth_request con 0x7f10ec06e0f0 auth_method 0 2024-03-25T18:22:54.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.598+0000 7f10fdffb640 10 client.4600.objecter ms_handle_connect 0x7f10ec06e0f0 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f11040546c0 con 0x7f11080a98f0 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 10 client.4600.objecter ms_dispatch 0x7f1108000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11040501d0 con 0x7f11080a98f0 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:54.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.605+0000 7f10fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:54.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.848+0000 7f110f6d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:54.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.848+0000 7f110f6d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:54.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.848+0000 7f110f6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:22:54.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.848+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11080627b0 con 0x7f11080a98f0 2024-03-25T18:22:54.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.886+0000 7f10fdffb640 1 -- 172.21.15.87:0/2504046072 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f1104052100 con 0x7f11080a98f0 2024-03-25T18:22:54.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.886+0000 7f10fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:54.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.886+0000 7f10fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:54.888 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:22:54.888 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.888+0000 7f110f6d6640 10 librados: watch_flush enter 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.888+0000 7f110f6d6640 10 librados: watch_flush exit 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.888+0000 7f110f6d6640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.888+0000 7f110f6d6640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 >> v1:172.21.15.171:6800/3417775508 conn(0x7f10ec06e0f0 legacy=0x7f10ec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 10 monclient: shutdown 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 >> v1:172.21.15.171:6789/0 conn(0x7f11080a98f0 legacy=0x7f11081dbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 shutdown_connections 2024-03-25T18:22:54.890 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 >> 172.21.15.87:0/2504046072 conn(0x7f11080b7ea0 msgr2=0x7f1108074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:54.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 shutdown_connections 2024-03-25T18:22:54.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 -- 172.21.15.87:0/2504046072 wait complete. 2024-03-25T18:22:54.891 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:22:54.889+0000 7f110f6d6640 1 librados: shutdown 2024-03-25T18:22:54.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:01.707 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:01.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.811+0000 7f0d8baa8640 10 monclient: get_monmap_and_config 2024-03-25T18:23:01.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.811+0000 7f0d8baa8640 10 monclient: build_initial_monmap 2024-03-25T18:23:01.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.811+0000 7f0d8baa8640 10 monclient: monmap: 2024-03-25T18:23:01.812 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:01.812251+0000 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:01.812251+0000 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:01.813 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:01.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.811+0000 7f0d8baa8640 1 Processor -- start 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 1 -- start start 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient: init 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d840b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d840b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8414be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840af940 con 0x7f0d840b4d60 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840b0a70 con 0x7f0d840b1730 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840b0d50 con 0x7f0d8414be90 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.812+0000 7f0d8baa8640 10 monclient(hunting): authenticate will time out at 8297.107809s 2024-03-25T18:23:01.815 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.813+0000 7f0d89043640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0d840b1730 0x7f0d840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:46952/0 (socket says 172.21.15.87:46952) 2024-03-25T18:23:01.816 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.813+0000 7f0d89043640 1 -- 172.21.15.87:0/2607403960 learned_addr learned my addr 172.21.15.87:0/2607403960 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141123214 0 0) 0x7f0d840af940 con 0x7f0d840b4d60 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 10 monclient(hunting): _negotiate 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:23:01.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.841+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70003520 con 0x7f0d840b4d60 2024-03-25T18:23:01.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582331912 0 0) 0x7f0d70003520 con 0x7f0d840b4d60 2024-03-25T18:23:01.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 >> v1:172.21.15.87:6790/0 conn(0x7f0d8414be90 legacy=0x7f0d8414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:01.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 >> v1:172.21.15.171:6789/0 conn(0x7f0d840b1730 legacy=0x7f0d840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:01.872 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: _finish_hunting 0 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 monclient: found mon.noname-a 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840b1030 con 0x7f0d840b4d60 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: _finish_auth 0 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: _check_auth_tickets 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d7c001690 con 0x7f0d840b4d60 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d8baa8640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d8baa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d88842640 10 monclient: dump: 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:01.873 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:01.874 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.871+0000 7f0d8baa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:01.882 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d7c001860 con 0x7f0d840b4d60 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 1 -- 172.21.15.87:0/2607403960 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d7c001cb0 con 0x7f0d840b4d60 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d8baa8640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d8baa8640 4 set_mon_vals no callback set 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.881+0000 7f0d88842640 10 monclient: dump: 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.883 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:01.884 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:01.884 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:01.884 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:01.884 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:01.884 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:01.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.882+0000 7f0d8baa8640 10 monclient: shutdown 2024-03-25T18:23:01.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.882+0000 7f0d8baa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:01.885 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.882+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2607403960 >> v1:172.21.15.87:6789/0 conn(0x7f0d840b4d60 legacy=0x7f0d8413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.882+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2607403960 shutdown_connections 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.882+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2607403960 >> 172.21.15.87:0/2607403960 conn(0x7f0d840617a0 msgr2=0x7f0d84061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.883+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2607403960 shutdown_connections 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.883+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2607403960 wait complete. 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.883+0000 7f0d8baa8640 10 monclient: build_initial_monmap 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.883+0000 7f0d8baa8640 10 monclient: monmap: 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:01.886 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 librados: starting msgr at 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 librados: starting objecter 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 Processor -- start 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 -- start start 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 librados: setting wanted keys 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 librados: calling monclient init 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient: init 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d840b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d840b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:01.887 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8414be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:01.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d84137fe0 con 0x7f0d840b1730 2024-03-25T18:23:01.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d84132520 con 0x7f0d8414be90 2024-03-25T18:23:01.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d841326b0 con 0x7f0d840b4d60 2024-03-25T18:23:01.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:01.888 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.884+0000 7f0d8baa8640 10 monclient(hunting): authenticate will time out at 8297.179814s 2024-03-25T18:23:01.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.891+0000 7f0d89844640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f0d840b1730 0x7f0d841377c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:53140/0 (socket says 172.21.15.87:53140) 2024-03-25T18:23:01.892 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.891+0000 7f0d89844640 1 -- 172.21.15.87:0/2574232416 learned_addr learned my addr 172.21.15.87:0/2574232416 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:01.896 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.895+0000 7f0d7a7fc640 10 client.?.objecter ms_handle_connect 0x7f0d840b4d60 2024-03-25T18:23:01.896 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.895+0000 7f0d7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:01.898 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.897+0000 7f0d7a7fc640 10 client.?.objecter ms_handle_connect 0x7f0d840b1730 2024-03-25T18:23:01.899 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.897+0000 7f0d7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620628545 0 0) 0x7f0d841326b0 con 0x7f0d840b4d60 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:23:01.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.903+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d68004eb0 con 0x7f0d840b4d60 2024-03-25T18:23:01.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.907+0000 7f0d7a7fc640 10 client.?.objecter ms_handle_connect 0x7f0d8414be90 2024-03-25T18:23:01.908 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.907+0000 7f0d7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303005792 0 0) 0x7f0d68004eb0 con 0x7f0d840b4d60 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d841326b0 con 0x7f0d840b4d60 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d740018a0 con 0x7f0d840b4d60 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:01.913 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:01.914 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.912+0000 7f0d7a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:01.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551158441 0 0) 0x7f0d84137fe0 con 0x7f0d840b1730 2024-03-25T18:23:01.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:01.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:01.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:01.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:23:01.923 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.921+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d68004eb0 con 0x7f0d840b1730 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242654383 0 0) 0x7f0d84132520 con 0x7f0d8414be90 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:23:01.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.924+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d84137fe0 con 0x7f0d8414be90 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.934+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2853963902 0 0) 0x7f0d841326b0 con 0x7f0d840b4d60 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 >> v1:172.21.15.171:6789/0 conn(0x7f0d8414be90 legacy=0x7f0d841d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 >> v1:172.21.15.87:6789/0 conn(0x7f0d840b1730 legacy=0x7f0d841377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 1 monclient: found mon.c 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:01.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84132840 con 0x7f0d840b4d60 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 10 monclient: _renew_subs 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d841408d0 con 0x7f0d840b4d60 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 10 monclient: _renew_subs 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d84144c20 con 0x7f0d840b4d60 2024-03-25T18:23:01.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.935+0000 7f0d8baa8640 1 librados: init done 2024-03-25T18:23:01.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.936+0000 7f0d8baa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:01.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.936+0000 7f0d8baa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:01.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.936+0000 7f0d8baa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:01.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:01.936+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d840627b0 con 0x7f0d840b4d60 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d74001d50 con 0x7f0d840b4d60 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0d74004980 con 0x7f0d840b4d60 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7a7fc640 10 monclient: dump: 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:02.243 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.241+0000 7f0d7bfff640 4 set_mon_vals no callback set 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.242+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f0d74004ba0 con 0x7f0d840b4d60 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.243+0000 7f0d89844640 10 monclient: get_auth_request con 0x7f0d6806de50 auth_method 0 2024-03-25T18:23:02.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.243+0000 7f0d7a7fc640 10 client.4607.objecter ms_handle_connect 0x7f0d6806de50 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f0d740547a0 con 0x7f0d840b4d60 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 10 client.4607.objecter ms_dispatch 0x7f0d84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d740500e0 con 0x7f0d840b4d60 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:02.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.248+0000 7f0d7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:02.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.490+0000 7f0d8baa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:02.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.490+0000 7f0d8baa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:02.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.490+0000 7f0d8baa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:02.492 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.490+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d84069f40 con 0x7f0d840b4d60 2024-03-25T18:23:02.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.531+0000 7f0d7a7fc640 1 -- 172.21.15.87:0/2574232416 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f0d740521e0 con 0x7f0d840b4d60 2024-03-25T18:23:02.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.531+0000 7f0d7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:02.532 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.531+0000 7f0d7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:02.532 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:02.532 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 10 librados: watch_flush enter 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 10 librados: watch_flush exit 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 10 client.4607.objecter successfully canceled tick 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 >> v1:172.21.15.171:6800/3417775508 conn(0x7f0d6806de50 legacy=0x7f0d68070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 10 monclient: shutdown 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:02.535 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 >> v1:172.21.15.87:6790/0 conn(0x7f0d840b4d60 legacy=0x7f0d84131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:02.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 shutdown_connections 2024-03-25T18:23:02.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.533+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 >> 172.21.15.87:0/2574232416 conn(0x7f0d840617a0 msgr2=0x7f0d840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:02.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.534+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 shutdown_connections 2024-03-25T18:23:02.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.534+0000 7f0d8baa8640 1 -- 172.21.15.87:0/2574232416 wait complete. 2024-03-25T18:23:02.536 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:02.534+0000 7f0d8baa8640 1 librados: shutdown 2024-03-25T18:23:02.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:09.355 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:09.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.467+0000 7f96dd6d6640 10 monclient: get_monmap_and_config 2024-03-25T18:23:09.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.468+0000 7f96dd6d6640 10 monclient: build_initial_monmap 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.468+0000 7f96dd6d6640 10 monclient: monmap: 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:09.469880+0000 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:09.469880+0000 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:09.471 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:09.472 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.468+0000 7f96dd6d6640 1 Processor -- start 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.468+0000 7f96dd6d6640 1 -- start start 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.468+0000 7f96dd6d6640 10 monclient: init 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f96d80a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f96d813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f96d81430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d8142520 con 0x7f96d81430c0 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d80b28a0 con 0x7f96d813f710 2024-03-25T18:23:09.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d80b2a30 con 0x7f96d80a98f0 2024-03-25T18:23:09.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:09.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.469+0000 7f96dd6d6640 10 monclient(hunting): authenticate will time out at 8304.765827s 2024-03-25T18:23:09.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.471+0000 7f96d6ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f96d80a98f0 0x7f96d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40084/0 (socket says 172.21.15.87:40084) 2024-03-25T18:23:09.474 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.471+0000 7f96d6ffd640 1 -- 172.21.15.87:0/2211316833 learned_addr learned my addr 172.21.15.87:0/2211316833 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:09.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920988379 0 0) 0x7f96d80b2a30 con 0x7f96d80a98f0 2024-03-25T18:23:09.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:09.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:09.481 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:09.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:23:09.482 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.479+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c0003520 con 0x7f96d80a98f0 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878448357 0 0) 0x7f96d80b28a0 con 0x7f96d813f710 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:23:09.491 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.489+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d80b2a30 con 0x7f96d813f710 2024-03-25T18:23:09.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280133155 0 0) 0x7f96c0003520 con 0x7f96d80a98f0 2024-03-25T18:23:09.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 >> v1:172.21.15.171:6789/0 conn(0x7f96d813f710 legacy=0x7f96d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:09.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 >> v1:172.21.15.87:6789/0 conn(0x7f96d81430c0 legacy=0x7f96d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:09.497 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d80b44c0 con 0x7f96d80a98f0 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f96c8001690 con 0x7f96d80a98f0 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96dd6d6640 5 monclient: authenticate success, global_id 4610 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96dd6d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:09.498 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96d5ffb640 10 monclient: dump: 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:09.499 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:09.500 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.500 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.495+0000 7f96dd6d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96c80017f0 con 0x7f96d80a98f0 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 1 -- 172.21.15.87:0/2211316833 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f96c8001c00 con 0x7f96d80a98f0 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 4 set_mon_vals no callback set 2024-03-25T18:23:09.503 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96d5ffb640 10 monclient: dump: 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:09.504 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:09.505 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:09.505 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:09.505 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 10 monclient: shutdown 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 1 -- 172.21.15.87:0/2211316833 >> v1:172.21.15.87:6790/0 conn(0x7f96d80a98f0 legacy=0x7f96d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.501+0000 7f96dd6d6640 1 -- 172.21.15.87:0/2211316833 shutdown_connections 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.502+0000 7f96dd6d6640 1 -- 172.21.15.87:0/2211316833 >> 172.21.15.87:0/2211316833 conn(0x7f96d80b7ea0 msgr2=0x7f96d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.502+0000 7f96dd6d6640 1 -- 172.21.15.87:0/2211316833 shutdown_connections 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.502+0000 7f96dd6d6640 1 -- 172.21.15.87:0/2211316833 wait complete. 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.502+0000 7f96dd6d6640 10 monclient: build_initial_monmap 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.502+0000 7f96dd6d6640 10 monclient: monmap: 2024-03-25T18:23:09.508 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 librados: starting msgr at 2024-03-25T18:23:09.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 librados: starting objecter 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 Processor -- start 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 -- start start 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 librados: setting wanted keys 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 1 librados: calling monclient init 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 10 monclient: init 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.503+0000 7f96dd6d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f96d80a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f96d813f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:09.510 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f96d81430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:09.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d813b2b0 con 0x7f96d81430c0 2024-03-25T18:23:09.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d813b500 con 0x7f96d813f710 2024-03-25T18:23:09.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d80b6f40 con 0x7f96d80a98f0 2024-03-25T18:23:09.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:09.511 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.504+0000 7f96dd6d6640 10 monclient(hunting): authenticate will time out at 8304.800087s 2024-03-25T18:23:09.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.514+0000 7f96d67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f96d813f710 0x7f96d81df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:33148/0 (socket says 172.21.15.87:33148) 2024-03-25T18:23:09.516 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.514+0000 7f96d67fc640 1 -- 172.21.15.87:0/1209151067 learned_addr learned my addr 172.21.15.87:0/1209151067 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:09.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.515+0000 7f96bf7fe640 10 client.?.objecter ms_handle_connect 0x7f96d813f710 2024-03-25T18:23:09.517 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.515+0000 7f96bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:09.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 client.?.objecter ms_handle_connect 0x7f96d80a98f0 2024-03-25T18:23:09.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:09.527 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216056537 0 0) 0x7f96d813b500 con 0x7f96d813f710 2024-03-25T18:23:09.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:09.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:09.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:09.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:23:09.528 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.525+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c4004eb0 con 0x7f96d813f710 2024-03-25T18:23:09.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.531+0000 7f96bf7fe640 10 client.?.objecter ms_handle_connect 0x7f96d81430c0 2024-03-25T18:23:09.533 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.531+0000 7f96bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:09.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891280457 0 0) 0x7f96d80b6f40 con 0x7f96d80a98f0 2024-03-25T18:23:09.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:09.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:09.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:09.542 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:23:09.543 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.540+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d813b500 con 0x7f96d80a98f0 2024-03-25T18:23:09.548 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502382716 0 0) 0x7f96c4004eb0 con 0x7f96d813f710 2024-03-25T18:23:09.548 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d80b6f40 con 0x7f96d813f710 2024-03-25T18:23:09.548 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f96cc001a90 con 0x7f96d813f710 2024-03-25T18:23:09.548 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:09.548 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.549 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.546+0000 7f96bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:09.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848211463 0 0) 0x7f96d813b2b0 con 0x7f96d81430c0 2024-03-25T18:23:09.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:09.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:09.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:09.550 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:23:09.551 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.547+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96c4004eb0 con 0x7f96d81430c0 2024-03-25T18:23:09.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1662319093 0 0) 0x7f96d80b6f40 con 0x7f96d813f710 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 >> v1:172.21.15.87:6790/0 conn(0x7f96d80a98f0 legacy=0x7f96d8139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 >> v1:172.21.15.87:6789/0 conn(0x7f96d81430c0 legacy=0x7f96d81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 1 monclient: found mon.b 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d813a430 con 0x7f96d813f710 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:09.561 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 10 monclient: _renew_subs 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d812dd10 con 0x7f96d813f710 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 10 monclient: _renew_subs 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d80b7b40 con 0x7f96d813f710 2024-03-25T18:23:09.562 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.558+0000 7f96dd6d6640 1 librados: init done 2024-03-25T18:23:09.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.559+0000 7f96dd6d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:09.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.559+0000 7f96dd6d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:09.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.559+0000 7f96dd6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:09.563 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.559+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d80627b0 con 0x7f96d813f710 2024-03-25T18:23:09.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96cc001cd0 con 0x7f96d813f710 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f96cc003a60 con 0x7f96d813f710 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96bf7fe640 10 monclient: dump: 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:09.854 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.851+0000 7f96d57fa640 4 set_mon_vals no callback set 2024-03-25T18:23:09.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.852+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f96cc003c80 con 0x7f96d813f710 2024-03-25T18:23:09.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.852+0000 7f96d6ffd640 10 monclient: get_auth_request con 0x7f96c406de00 auth_method 0 2024-03-25T18:23:09.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.853+0000 7f96bf7fe640 10 client.4612.objecter ms_handle_connect 0x7f96c406de00 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.858+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f96cc0548b0 con 0x7f96d813f710 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.858+0000 7f96bf7fe640 10 client.4612.objecter ms_dispatch 0x7f96d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.858+0000 7f96bf7fe640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.858+0000 7f96bf7fe640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.858+0000 7f96bf7fe640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.859+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96cc0503c0 con 0x7f96d813f710 2024-03-25T18:23:09.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.859+0000 7f96bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:09.862 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:09.859+0000 7f96bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:10.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.108+0000 7f96dd6d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:10.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.108+0000 7f96dd6d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:10.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.108+0000 7f96dd6d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:10.111 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.108+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96d81434a0 con 0x7f96d813f710 2024-03-25T18:23:10.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.142+0000 7f96bf7fe640 1 -- 172.21.15.87:0/1209151067 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f96cc0522f0 con 0x7f96d813f710 2024-03-25T18:23:10.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.142+0000 7f96bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:10.144 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.142+0000 7f96bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:10.144 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:10.145 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:10.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 10 librados: watch_flush enter 2024-03-25T18:23:10.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 10 librados: watch_flush exit 2024-03-25T18:23:10.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:23:10.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:23:10.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 >> v1:172.21.15.171:6800/3417775508 conn(0x7f96c406de00 legacy=0x7f96c40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 10 monclient: shutdown 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 >> v1:172.21.15.171:6789/0 conn(0x7f96d813f710 legacy=0x7f96d81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 shutdown_connections 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 >> 172.21.15.87:0/1209151067 conn(0x7f96d80b7ea0 msgr2=0x7f96d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 shutdown_connections 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 -- 172.21.15.87:0/1209151067 wait complete. 2024-03-25T18:23:10.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:10.144+0000 7f96dd6d6640 1 librados: shutdown 2024-03-25T18:23:10.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:13.019 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:00.733 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.026 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:00.740 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.051 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:00.765 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.078 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:00.756 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.083 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:00.796 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.085 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:00.763 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.091 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:00.804 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.115 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:00.794 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.129 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:00.842 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:16.965 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 10 monclient: get_monmap_and_config 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 10 monclient: build_initial_monmap 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 10 monclient: monmap: 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:17.081928+0000 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:17.081928+0000 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:17.082 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:17.083 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 1 Processor -- start 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 1 -- start start 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.081+0000 7f2791c63640 10 monclient: init 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-b con 0x7f278c0b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-a con 0x7f278c0b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-c con 0x7f278c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0af940 con 0x7f278c0b1730 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0b0a70 con 0x7f278c14be90 2024-03-25T18:23:17.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0b0d50 con 0x7f278c0b4d60 2024-03-25T18:23:17.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:17.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.082+0000 7f2791c63640 10 monclient(hunting): authenticate will time out at 8312.377556s 2024-03-25T18:23:17.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.085+0000 7f278b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f278c0b4d60 0x7f278c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:44660/0 (socket says 172.21.15.87:44660) 2024-03-25T18:23:17.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.085+0000 7f278b7fe640 1 -- 172.21.15.87:0/3319312547 learned_addr learned my addr 172.21.15.87:0/3319312547 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230794731 0 0) 0x7f278c0b0d50 con 0x7f278c0b4d60 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:23:17.100 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.099+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774003520 con 0x7f278c0b4d60 2024-03-25T18:23:17.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.112+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442834146 0 0) 0x7f278c0b0a70 con 0x7f278c14be90 2024-03-25T18:23:17.113 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.112+0000 7f278a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.112+0000 7f278a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.112+0000 7f278a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.113+0000 7f278a7fc640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:23:17.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.113+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c0b0d50 con 0x7f278c14be90 2024-03-25T18:23:17.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561429294 0 0) 0x7f278c0af940 con 0x7f278c0b1730 2024-03-25T18:23:17.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.131 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:23:17.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.130+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c0b0a70 con 0x7f278c0b1730 2024-03-25T18:23:17.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598757519 0 0) 0x7f2774003520 con 0x7f278c0b4d60 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 >> v1:172.21.15.171:6789/0 conn(0x7f278c14be90 legacy=0x7f278c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 >> v1:172.21.15.87:6789/0 conn(0x7f278c0b1730 legacy=0x7f278c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0b1030 con 0x7f278c0b4d60 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f277c001690 con 0x7f278c0b4d60 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f2791c63640 5 monclient: authenticate success, global_id 4616 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f2791c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:17.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f278a7fc640 10 monclient: dump: 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.134 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.131+0000 7f2791c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:17.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f277c0017f0 con 0x7f278c0b4d60 2024-03-25T18:23:17.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:17.160 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 1 -- 172.21.15.87:0/3319312547 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f277c001c00 con 0x7f278c0b4d60 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f278a7fc640 10 monclient: dump: 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.161 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f2791c63640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:17.162 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.159+0000 7f2791c63640 4 set_mon_vals no callback set 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 10 monclient: shutdown 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 1 -- 172.21.15.87:0/3319312547 >> v1:172.21.15.87:6790/0 conn(0x7f278c0b4d60 legacy=0x7f278c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 1 -- 172.21.15.87:0/3319312547 shutdown_connections 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 1 -- 172.21.15.87:0/3319312547 >> 172.21.15.87:0/3319312547 conn(0x7f278c0617a0 msgr2=0x7f278c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 1 -- 172.21.15.87:0/3319312547 shutdown_connections 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.160+0000 7f2791c63640 1 -- 172.21.15.87:0/3319312547 wait complete. 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.161+0000 7f2791c63640 10 monclient: build_initial_monmap 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.161+0000 7f2791c63640 10 monclient: monmap: 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:17.164 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.161+0000 7f2791c63640 1 librados: starting msgr at 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.161+0000 7f2791c63640 1 librados: starting objecter 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 Processor -- start 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 -- start start 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 librados: setting wanted keys 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 librados: calling monclient init 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient: init 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:17.165 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-c con 0x7f278c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-b con 0x7f278c0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient(hunting): picked mon.noname-a con 0x7f278c14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c1d7980 con 0x7f278c14be90 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c1420a0 con 0x7f278c0b4d60 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c1432c0 con 0x7f278c0b1730 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:17.166 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.162+0000 7f2791c63640 10 monclient(hunting): authenticate will time out at 8312.457831s 2024-03-25T18:23:17.168 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.167+0000 7f278b7fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f278c0b1730 0x7f278c141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:44752/0 (socket says 172.21.15.87:44752) 2024-03-25T18:23:17.168 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.167+0000 7f278b7fe640 1 -- 172.21.15.87:0/4264683813 learned_addr learned my addr 172.21.15.87:0/4264683813 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:17.170 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.169+0000 7f2773fff640 10 client.?.objecter ms_handle_connect 0x7f278c0b4d60 2024-03-25T18:23:17.170 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.169+0000 7f2773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:17.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.170+0000 7f2773fff640 10 client.?.objecter ms_handle_connect 0x7f278c0b1730 2024-03-25T18:23:17.171 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.170+0000 7f2773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:17.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.242+0000 7f2773fff640 10 client.?.objecter ms_handle_connect 0x7f278c14be90 2024-03-25T18:23:17.243 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.242+0000 7f2773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145804857 0 0) 0x7f278c1432c0 con 0x7f278c0b1730 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:23:17.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.248+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2778004eb0 con 0x7f278c0b1730 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.252+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262030109 0 0) 0x7f278c1420a0 con 0x7f278c0b4d60 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.253+0000 7f2773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.253+0000 7f2773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.253+0000 7f2773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.253+0000 7f2773fff640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:23:17.254 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.253+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c1432c0 con 0x7f278c0b4d60 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971406825 0 0) 0x7f278c1d7980 con 0x7f278c14be90 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:23:17.258 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.257+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c1420a0 con 0x7f278c14be90 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3523626978 0 0) 0x7f2778004eb0 con 0x7f278c0b1730 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f278c1d7980 con 0x7f278c0b1730 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f277c001bc0 con 0x7f278c0b1730 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 10 monclient(hunting): dump: 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:17.269 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:17.270 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:17.270 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.270 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.267+0000 7f2773fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087057861 0 0) 0x7f278c1432c0 con 0x7f278c0b4d60 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2778004eb0 con 0x7f278c0b4d60 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2780001870 con 0x7f278c0b4d60 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.271+0000 7f2773fff640 10 monclient(hunting): dump: 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:17.273 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:17.274 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:17.274 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2966331327 0 0) 0x7f2778004eb0 con 0x7f278c0b4d60 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 >> v1:172.21.15.87:6790/0 conn(0x7f278c0b1730 legacy=0x7f278c141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 >> v1:172.21.15.87:6789/0 conn(0x7f278c14be90 legacy=0x7f278c1d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 1 monclient: found mon.b 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c143940 con 0x7f278c0b4d60 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:23:17.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 10 monclient: _renew_subs 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c138a70 con 0x7f278c0b4d60 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 10 monclient: _renew_subs 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c1d8700 con 0x7f278c0b4d60 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.279+0000 7f2791c63640 1 librados: init done 2024-03-25T18:23:17.282 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.280+0000 7f2791c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:17.283 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.280+0000 7f2791c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:17.283 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.280+0000 7f2791c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:17.283 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.280+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c143130 con 0x7f278c0b4d60 2024-03-25T18:23:17.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.284+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27800035f0 con 0x7f278c0b4d60 2024-03-25T18:23:17.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.284+0000 7f2773fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:17.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.284+0000 7f2789ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.556+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2780001d80 con 0x7f278c0b4d60 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.556+0000 7f2773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.556+0000 7f2773fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.556+0000 7f2773fff640 10 monclient: dump: 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:17.557 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:17.558 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:17.566 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.565+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2780003ac0 con 0x7f278c0b4d60 2024-03-25T18:23:17.566 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.565+0000 7f278b7fe640 10 monclient: get_auth_request con 0x7f277806dfb0 auth_method 0 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 10 client.4618.objecter ms_handle_connect 0x7f277806dfb0 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f27800535f0 con 0x7f278c0b4d60 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 10 client.4618.objecter ms_dispatch 0x7f278c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:17.567 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.566+0000 7f2773fff640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:23:17.569 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.568+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f278004f100 con 0x7f278c0b4d60 2024-03-25T18:23:17.569 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.568+0000 7f2773fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:17.569 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.568+0000 7f2773fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:17.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.824+0000 7f2791c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:17.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.824+0000 7f2791c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:17.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.824+0000 7f2791c63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:17.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.824+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278c1401f0 con 0x7f278c0b4d60 2024-03-25T18:23:17.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.851+0000 7f2773fff640 1 -- 172.21.15.87:0/4264683813 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f2780004480 con 0x7f278c0b4d60 2024-03-25T18:23:17.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.851+0000 7f2773fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:17.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.851+0000 7f2773fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:17.852 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:17.853 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 10 librados: watch_flush enter 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 10 librados: watch_flush exit 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 >> v1:172.21.15.171:6800/3417775508 conn(0x7f277806dfb0 legacy=0x7f2778070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 10 monclient: shutdown 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 >> v1:172.21.15.171:6789/0 conn(0x7f278c0b4d60 legacy=0x7f278c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 shutdown_connections 2024-03-25T18:23:17.855 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.853+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 >> 172.21.15.87:0/4264683813 conn(0x7f278c0617a0 msgr2=0x7f278c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:17.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.854+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 shutdown_connections 2024-03-25T18:23:17.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.854+0000 7f2791c63640 1 -- 172.21.15.87:0/4264683813 wait complete. 2024-03-25T18:23:17.856 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:17.854+0000 7f2791c63640 1 librados: shutdown 2024-03-25T18:23:17.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:24.674 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:24.795 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 10 monclient: build_initial_monmap 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 10 monclient: monmap: 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:24.795877+0000 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:24.795877+0000 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:24.796 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:24.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 1 Processor -- start 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 1 -- start start 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.795+0000 7f3fe9d0e640 10 monclient: init 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe40a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe41430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe4142520 con 0x7f3fe40a98f0 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40b28a0 con 0x7f3fe41430c0 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40b2a30 con 0x7f3fe413f710 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:24.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.796+0000 7f3fe9d0e640 10 monclient(hunting): authenticate will time out at 8320.091803s 2024-03-25T18:23:24.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.808+0000 7f3fe3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3fe41430c0 0x7f3fe414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:48148/0 (socket says 172.21.15.87:48148) 2024-03-25T18:23:24.809 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.808+0000 7f3fe3fff640 1 -- 172.21.15.87:0/2551682206 learned_addr learned my addr 172.21.15.87:0/2551682206 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800243486 0 0) 0x7f3fe40b2a30 con 0x7f3fe413f710 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:23:24.825 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.824+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fcc003520 con 0x7f3fe413f710 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.825+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946136534 0 0) 0x7f3fe40b28a0 con 0x7f3fe41430c0 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.826+0000 7f3fe27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.826+0000 7f3fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.826+0000 7f3fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.826+0000 7f3fe27fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:23:24.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.826+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe40b2a30 con 0x7f3fe41430c0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510729993 0 0) 0x7f3fe40b2a30 con 0x7f3fe41430c0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 >> v1:172.21.15.87:6790/0 conn(0x7f3fe413f710 legacy=0x7f3fe41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 >> v1:172.21.15.87:6789/0 conn(0x7f3fe40a98f0 legacy=0x7f3fe40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40b44c0 con 0x7f3fe41430c0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:24.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f3fdc0019f0 con 0x7f3fe41430c0 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe27fc640 10 monclient: dump: 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:24.838 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:24.839 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:24.839 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:24.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe9d0e640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:23:24.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.836+0000 7f3fe9d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:24.843 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.842+0000 7f3fe27fc640 1 -- 172.21.15.87:0/2551682206 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fdc001b50 con 0x7f3fe41430c0 2024-03-25T18:23:24.843 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.842+0000 7f3fe27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:24.843 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.842+0000 7f3fe9d0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:24.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.842+0000 7f3fe9d0e640 4 set_mon_vals no callback set 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 10 monclient: shutdown 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/2551682206 >> v1:172.21.15.171:6789/0 conn(0x7f3fe41430c0 legacy=0x7f3fe414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/2551682206 shutdown_connections 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/2551682206 >> 172.21.15.87:0/2551682206 conn(0x7f3fe40b7ea0 msgr2=0x7f3fe40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/2551682206 shutdown_connections 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.843+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/2551682206 wait complete. 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.844+0000 7f3fe9d0e640 10 monclient: build_initial_monmap 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.844+0000 7f3fe9d0e640 10 monclient: monmap: 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:24.845 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:24.846 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:24.847 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.844+0000 7f3fe9d0e640 1 librados: starting msgr at 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.844+0000 7f3fe9d0e640 1 librados: starting objecter 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.844+0000 7f3fe9d0e640 1 Processor -- start 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 1 -- start start 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 1 librados: setting wanted keys 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 1 librados: calling monclient init 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 10 monclient: init 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.845+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40abd10 con 0x7f3fe41430c0 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40ac260 con 0x7f3fe40a98f0 2024-03-25T18:23:24.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40abf80 con 0x7f3fe413f710 2024-03-25T18:23:24.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:24.849 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.846+0000 7f3fe9d0e640 10 monclient(hunting): authenticate will time out at 8320.141030s 2024-03-25T18:23:24.851 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.850+0000 7f3fe3fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f3fe41430c0 0x7f3fe41e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:54236/0 (socket says 172.21.15.87:54236) 2024-03-25T18:23:24.852 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.850+0000 7f3fe3fff640 1 -- 172.21.15.87:0/1213569964 learned_addr learned my addr 172.21.15.87:0/1213569964 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:24.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.852+0000 7f3fcbfff640 10 client.?.objecter ms_handle_connect 0x7f3fe41430c0 2024-03-25T18:23:24.853 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.852+0000 7f3fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:24.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.858+0000 7f3fcbfff640 10 client.?.objecter ms_handle_connect 0x7f3fe413f710 2024-03-25T18:23:24.859 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.858+0000 7f3fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414051374 0 0) 0x7f3fe40abd10 con 0x7f3fe41430c0 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:23:24.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.859+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0004eb0 con 0x7f3fe41430c0 2024-03-25T18:23:24.862 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.861+0000 7f3fcbfff640 10 client.?.objecter ms_handle_connect 0x7f3fe40a98f0 2024-03-25T18:23:24.862 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.861+0000 7f3fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:24.868 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2344692354 0 0) 0x7f3fd0004eb0 con 0x7f3fe41430c0 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe40abd10 con 0x7f3fe41430c0 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f3fdc001f60 con 0x7f3fe41430c0 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 10 monclient(hunting): dump: 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:24.869 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:24.870 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.867+0000 7f3fcbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:24.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3853989896 0 0) 0x7f3fe40abd10 con 0x7f3fe41430c0 2024-03-25T18:23:24.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 >> v1:172.21.15.87:6790/0 conn(0x7f3fe413f710 legacy=0x7f3fe40ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:24.877 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 >> v1:172.21.15.171:6789/0 conn(0x7f3fe40a98f0 legacy=0x7f3fe4138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 1 monclient: found mon.a 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4139310 con 0x7f3fe41430c0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 10 monclient: _finish_auth 0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 10 monclient: _renew_subs 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4137f70 con 0x7f3fe41430c0 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 10 client.4614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:24.878 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 10 monclient: _renew_subs 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe4136930 con 0x7f3fe41430c0 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.876+0000 7f3fe9d0e640 1 librados: init done 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.877+0000 7f3fe9d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.877+0000 7f3fe9d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.877+0000 7f3fe9d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:23:24.879 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:24.877+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4138160 con 0x7f3fe41430c0 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fdc004e10 con 0x7f3fe41430c0 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f3fdc005120 con 0x7f3fe41430c0 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fcbfff640 10 monclient: dump: 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:25.132 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.130+0000 7f3fe1ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.131+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f3fdc01b0f0 con 0x7f3fe41430c0 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.132+0000 7f3fe37fe640 10 monclient: get_auth_request con 0x7f3fd006dd00 auth_method 0 2024-03-25T18:23:25.133 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.132+0000 7f3fcbfff640 10 client.4614.objecter ms_handle_connect 0x7f3fd006dd00 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f3fdc054be0 con 0x7f3fe41430c0 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 10 client.4614.objecter ms_dispatch 0x7f3fe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 3 client.4614.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 3 client.4614.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 20 client.4614.objecter dump_active .. 0 homeless 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fdc0506f0 con 0x7f3fe41430c0 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:25.137 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.136+0000 7f3fcbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:25.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.375+0000 7f3fe9d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:25.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.375+0000 7f3fe9d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:25.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.375+0000 7f3fe9d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:23:25.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.375+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe4136c30 con 0x7f3fe41430c0 2024-03-25T18:23:25.401 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.400+0000 7f3fcbfff640 1 -- 172.21.15.87:0/1213569964 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f3fdc052620 con 0x7f3fe41430c0 2024-03-25T18:23:25.401 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.400+0000 7f3fcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:25.401 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.400+0000 7f3fcbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:25.401 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:25.401 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:25.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 10 librados: watch_flush enter 2024-03-25T18:23:25.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 10 librados: watch_flush exit 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 20 client.4614.objecter shutdown clearing up homeless session... 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 10 client.4614.objecter successfully canceled tick 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 >> v1:172.21.15.171:6800/3417775508 conn(0x7f3fd006dd00 legacy=0x7f3fd00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 10 monclient: shutdown 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 >> v1:172.21.15.87:6789/0 conn(0x7f3fe41430c0 legacy=0x7f3fe41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 shutdown_connections 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 >> 172.21.15.87:0/1213569964 conn(0x7f3fe40b7ea0 msgr2=0x7f3fe4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 shutdown_connections 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 -- 172.21.15.87:0/1213569964 wait complete. 2024-03-25T18:23:25.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:25.402+0000 7f3fe9d0e640 1 librados: shutdown 2024-03-25T18:23:25.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:32.220 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.332+0000 7f0b28482640 10 monclient: get_monmap_and_config 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.332+0000 7f0b28482640 10 monclient: build_initial_monmap 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.332+0000 7f0b28482640 10 monclient: monmap: 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:32.333024+0000 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:32.333024+0000 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:32.333 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.333+0000 7f0b28482640 1 Processor -- start 2024-03-25T18:23:32.334 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.333+0000 7f0b28482640 1 -- start start 2024-03-25T18:23:32.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.333+0000 7f0b28482640 10 monclient: init 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-b con 0x7f0b200a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-a con 0x7f0b201430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b20142520 con 0x7f0b201430c0 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b200b28a0 con 0x7f0b2013f710 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b200b2a30 con 0x7f0b200a98f0 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:32.336 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.334+0000 7f0b28482640 10 monclient(hunting): authenticate will time out at 8327.629390s 2024-03-25T18:23:32.341 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.341+0000 7f0b25a1d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0b2013f710 0x7f0b201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:36678/0 (socket says 172.21.15.87:36678) 2024-03-25T18:23:32.342 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.341+0000 7f0b25a1d640 1 -- 172.21.15.87:0/3129312589 learned_addr learned my addr 172.21.15.87:0/3129312589 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830405429 0 0) 0x7f0b200b28a0 con 0x7f0b2013f710 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 10 monclient(hunting): _negotiate 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:23:32.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.349+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c003520 con 0x7f0b2013f710 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859849556 0 0) 0x7f0b0c003520 con 0x7f0b2013f710 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 >> v1:172.21.15.87:6790/0 conn(0x7f0b200a98f0 legacy=0x7f0b200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 >> v1:172.21.15.87:6789/0 conn(0x7f0b201430c0 legacy=0x7f0b2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: _finish_hunting 0 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 monclient: found mon.noname-c 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200b44c0 con 0x7f0b2013f710 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: _finish_auth 0 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: _check_auth_tickets 2024-03-25T18:23:32.357 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b10001690 con 0x7f0b2013f710 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b2521c640 10 monclient: dump: 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.358 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b28482640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:23:32.359 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.355+0000 7f0b28482640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:32.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b10001870 con 0x7f0b2013f710 2024-03-25T18:23:32.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:32.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 1 -- 172.21.15.87:0/3129312589 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b10001cc0 con 0x7f0b2013f710 2024-03-25T18:23:32.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b2521c640 10 monclient: dump: 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b28482640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:32.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.366+0000 7f0b28482640 4 set_mon_vals no callback set 2024-03-25T18:23:32.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 10 monclient: shutdown 2024-03-25T18:23:32.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:32.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 1 -- 172.21.15.87:0/3129312589 >> v1:172.21.15.171:6789/0 conn(0x7f0b2013f710 legacy=0x7f0b201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:32.370 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 1 -- 172.21.15.87:0/3129312589 shutdown_connections 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 1 -- 172.21.15.87:0/3129312589 >> 172.21.15.87:0/3129312589 conn(0x7f0b200b7ea0 msgr2=0x7f0b200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 1 -- 172.21.15.87:0/3129312589 shutdown_connections 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.367+0000 7f0b28482640 1 -- 172.21.15.87:0/3129312589 wait complete. 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 10 monclient: build_initial_monmap 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 10 monclient: monmap: 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:32.371 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 librados: starting msgr at 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 librados: starting objecter 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 Processor -- start 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 -- start start 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 librados: setting wanted keys 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 1 librados: calling monclient init 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.368+0000 7f0b28482640 10 monclient: init 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-c con 0x7f0b200a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient(hunting): picked mon.noname-a con 0x7f0b201430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b20137720 con 0x7f0b201430c0 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b201dc070 con 0x7f0b2013f710 2024-03-25T18:23:32.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b20139140 con 0x7f0b200a98f0 2024-03-25T18:23:32.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:32.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.369+0000 7f0b28482640 10 monclient(hunting): authenticate will time out at 8327.664098s 2024-03-25T18:23:32.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.395+0000 7f0b26a1f640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f0b201430c0 0x7f0b201db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:55012/0 (socket says 172.21.15.87:55012) 2024-03-25T18:23:32.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.395+0000 7f0b26a1f640 1 -- 172.21.15.87:0/3278534643 learned_addr learned my addr 172.21.15.87:0/3278534643 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:32.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.396+0000 7f0b16ffd640 10 client.?.objecter ms_handle_connect 0x7f0b201430c0 2024-03-25T18:23:32.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.396+0000 7f0b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:32.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.399+0000 7f0b16ffd640 10 client.?.objecter ms_handle_connect 0x7f0b2013f710 2024-03-25T18:23:32.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.399+0000 7f0b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:32.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.399+0000 7f0b16ffd640 10 client.?.objecter ms_handle_connect 0x7f0b200a98f0 2024-03-25T18:23:32.400 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.399+0000 7f0b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:32.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909794200 0 0) 0x7f0b20137720 con 0x7f0b201430c0 2024-03-25T18:23:32.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:32.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:32.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:32.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:23:32.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.415+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b04004eb0 con 0x7f0b201430c0 2024-03-25T18:23:32.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285986572 0 0) 0x7f0b201dc070 con 0x7f0b2013f710 2024-03-25T18:23:32.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:32.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:32.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:32.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:23:32.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.417+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b20137720 con 0x7f0b2013f710 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318457085 0 0) 0x7f0b20139140 con 0x7f0b200a98f0 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:23:32.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.420+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b201dc070 con 0x7f0b200a98f0 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 861954095 0 0) 0x7f0b04004eb0 con 0x7f0b201430c0 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b20139140 con 0x7f0b201430c0 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b18001c30 con 0x7f0b201430c0 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.431 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.430+0000 7f0b16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.448+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460171146 0 0) 0x7f0b20137720 con 0x7f0b2013f710 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.449+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b04004eb0 con 0x7f0b2013f710 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.449+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b10001b80 con 0x7f0b2013f710 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.449+0000 7f0b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.449+0000 7f0b16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.449+0000 7f0b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.450 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.451 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.451 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998934285 0 0) 0x7f0b201dc070 con 0x7f0b200a98f0 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b20137720 con 0x7f0b200a98f0 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b1c001830 con 0x7f0b200a98f0 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.452+0000 7f0b16ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.454 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.455 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.455 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.455 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 973593062 0 0) 0x7f0b04004eb0 con 0x7f0b2013f710 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 >> v1:172.21.15.87:6790/0 conn(0x7f0b200a98f0 legacy=0x7f0b20137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 >> v1:172.21.15.87:6789/0 conn(0x7f0b201430c0 legacy=0x7f0b201db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 1 monclient: found mon.b 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20139420 con 0x7f0b2013f710 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:23:32.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 10 monclient: _renew_subs 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b20136dc0 con 0x7f0b2013f710 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 10 monclient: _renew_subs 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b20139840 con 0x7f0b2013f710 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.456+0000 7f0b28482640 1 librados: init done 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.458+0000 7f0b28482640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:32.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.458+0000 7f0b28482640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:32.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.458+0000 7f0b28482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:32.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.458+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2014f2d0 con 0x7f0b2013f710 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b10001ce0 con 0x7f0b2013f710 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0b10004d90 con 0x7f0b2013f710 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b16ffd640 10 monclient: dump: 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.740 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.739+0000 7f0b24a1b640 4 set_mon_vals no callback set 2024-03-25T18:23:32.741 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.740+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f0b1001ad60 con 0x7f0b2013f710 2024-03-25T18:23:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.740+0000 7f0b2621e640 10 monclient: get_auth_request con 0x7f0b0406e100 auth_method 0 2024-03-25T18:23:32.742 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.740+0000 7f0b16ffd640 10 client.4627.objecter ms_handle_connect 0x7f0b0406e100 2024-03-25T18:23:32.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f0b10054890 con 0x7f0b2013f710 2024-03-25T18:23:32.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 10 client.4627.objecter ms_dispatch 0x7f0b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:32.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:32.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:32.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:23:32.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.748+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b100503a0 con 0x7f0b2013f710 2024-03-25T18:23:32.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.749+0000 7f0b16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:32.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:32.749+0000 7f0b16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:33.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.021+0000 7f0b28482640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:33.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.021+0000 7f0b28482640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:33.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.021+0000 7f0b28482640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:33.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.021+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b20136f50 con 0x7f0b2013f710 2024-03-25T18:23:33.101 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:20.814 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.111 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:20.824 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.149 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:20.862 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.174 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:20.852 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.181 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:20.894 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.186 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:20.864 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.188 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:20.901 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.192 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.190+0000 7f0b16ffd640 1 -- 172.21.15.87:0/3278534643 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f0b100522d0 con 0x7f0b2013f710 2024-03-25T18:23:33.192 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.191+0000 7f0b16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:33.192 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.191+0000 7f0b16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:33.192 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:33.192 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 10 librados: watch_flush enter 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 10 librados: watch_flush exit 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 >> v1:172.21.15.171:6800/3417775508 conn(0x7f0b0406e100 legacy=0x7f0b040705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 10 monclient: shutdown 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 >> v1:172.21.15.171:6789/0 conn(0x7f0b2013f710 legacy=0x7f0b201db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 shutdown_connections 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 >> 172.21.15.87:0/3278534643 conn(0x7f0b200b7ea0 msgr2=0x7f0b20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:33.195 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 shutdown_connections 2024-03-25T18:23:33.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 -- 172.21.15.87:0/3278534643 wait complete. 2024-03-25T18:23:33.196 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:33.193+0000 7f0b28482640 1 librados: shutdown 2024-03-25T18:23:33.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:33.211 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:20.890 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.227 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:20.940 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:40.010 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:40.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.124+0000 7f53d261e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:40.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.124+0000 7f53d261e640 10 monclient: build_initial_monmap 2024-03-25T18:23:40.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.124+0000 7f53d261e640 10 monclient: monmap: 2024-03-25T18:23:40.125 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:40.125459+0000 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:40.125459+0000 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:40.126 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.125+0000 7f53d261e640 1 Processor -- start 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.125+0000 7f53d261e640 1 -- start start 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.125+0000 7f53d261e640 10 monclient: init 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:40.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-c con 0x7f53cc0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-b con 0x7f53cc0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-a con 0x7f53cc14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0af940 con 0x7f53cc14be90 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0b0a70 con 0x7f53cc0b4d60 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0b0d50 con 0x7f53cc0b1730 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:40.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.126+0000 7f53d261e640 10 monclient(hunting): authenticate will time out at 8335.421029s 2024-03-25T18:23:40.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.135+0000 7f53d0bbb640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f53cc14be90 0x7f53cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:50808/0 (socket says 172.21.15.87:50808) 2024-03-25T18:23:40.136 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.135+0000 7f53d0bbb640 1 -- 172.21.15.87:0/3620856607 learned_addr learned my addr 172.21.15.87:0/3620856607 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:40.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295596469 0 0) 0x7f53cc0af940 con 0x7f53cc14be90 2024-03-25T18:23:40.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:40.145 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:40.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:40.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:23:40.146 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.144+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b4003520 con 0x7f53cc14be90 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498224373 0 0) 0x7f53cc0b0a70 con 0x7f53cc0b4d60 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:23:40.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.152+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc0af940 con 0x7f53cc0b4d60 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391758071 0 0) 0x7f53b4003520 con 0x7f53cc14be90 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 >> v1:172.21.15.87:6790/0 conn(0x7f53cc0b1730 legacy=0x7f53cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 >> v1:172.21.15.171:6789/0 conn(0x7f53cc0b4d60 legacy=0x7f53cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 monclient: found mon.noname-a 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:40.157 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc0b1030 con 0x7f53cc14be90 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f53c4001a10 con 0x7f53cc14be90 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53d261e640 5 monclient: authenticate success, global_id 4623 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53d261e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53caffd640 10 monclient: dump: 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:40.158 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.159 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:40.159 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:40.159 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:40.159 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.159 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.156+0000 7f53d261e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c4001b70 con 0x7f53cc14be90 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 1 -- 172.21.15.87:0/3620856607 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f53c4001f80 con 0x7f53cc14be90 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53caffd640 10 monclient: dump: 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:40.177 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53d261e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:40.178 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53d261e640 4 set_mon_vals no callback set 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53d261e640 10 monclient: shutdown 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53d261e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.176+0000 7f53d261e640 1 -- 172.21.15.87:0/3620856607 >> v1:172.21.15.87:6789/0 conn(0x7f53cc14be90 legacy=0x7f53cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 1 -- 172.21.15.87:0/3620856607 shutdown_connections 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 1 -- 172.21.15.87:0/3620856607 >> 172.21.15.87:0/3620856607 conn(0x7f53cc0617a0 msgr2=0x7f53cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 1 -- 172.21.15.87:0/3620856607 shutdown_connections 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 1 -- 172.21.15.87:0/3620856607 wait complete. 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 10 monclient: build_initial_monmap 2024-03-25T18:23:40.179 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.177+0000 7f53d261e640 10 monclient: monmap: 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 librados: starting msgr at 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 librados: starting objecter 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 Processor -- start 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 -- start start 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 librados: setting wanted keys 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 1 librados: calling monclient init 2024-03-25T18:23:40.180 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 10 monclient: init 2024-03-25T18:23:40.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.178+0000 7f53d261e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:40.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:40.181 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-c con 0x7f53cc0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-b con 0x7f53cc0b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient(hunting): picked mon.noname-a con 0x7f53cc14be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc134a00 con 0x7f53cc14be90 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc139050 con 0x7f53cc0b4d60 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc13a330 con 0x7f53cc0b1730 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:40.182 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.179+0000 7f53d261e640 10 monclient(hunting): authenticate will time out at 8335.474023s 2024-03-25T18:23:40.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.182+0000 7f53d0bbb640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f53cc14be90 0x7f53cc1342f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:50842/0 (socket says 172.21.15.87:50842) 2024-03-25T18:23:40.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.182+0000 7f53d0bbb640 1 -- 172.21.15.87:0/1857600306 learned_addr learned my addr 172.21.15.87:0/1857600306 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:40.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.183+0000 7f53c8ff9640 10 client.?.objecter ms_handle_connect 0x7f53cc14be90 2024-03-25T18:23:40.183 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.183+0000 7f53c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:40.193 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.192+0000 7f53c8ff9640 10 client.?.objecter ms_handle_connect 0x7f53cc0b4d60 2024-03-25T18:23:40.193 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.192+0000 7f53c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:40.199 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.198+0000 7f53c8ff9640 10 client.?.objecter ms_handle_connect 0x7f53cc0b1730 2024-03-25T18:23:40.199 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.198+0000 7f53c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:40.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095135015 0 0) 0x7f53cc134a00 con 0x7f53cc14be90 2024-03-25T18:23:40.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:40.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:40.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:40.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.200+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8004eb0 con 0x7f53cc14be90 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281522958 0 0) 0x7f53cc139050 con 0x7f53cc0b4d60 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:40.202 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:23:40.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.201+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc134a00 con 0x7f53cc0b4d60 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839950552 0 0) 0x7f53cc134a00 con 0x7f53cc0b4d60 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53cc139050 con 0x7f53cc0b4d60 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f53c00018b0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:40.215 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.213+0000 7f53c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550380954 0 0) 0x7f53cc13a330 con 0x7f53cc0b1730 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:40.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.214+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc134a00 con 0x7f53cc0b1730 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107311875 0 0) 0x7f53b8004eb0 con 0x7f53cc14be90 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53cc13a330 con 0x7f53cc14be90 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f53c4001ec0 con 0x7f53cc14be90 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:40.217 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.216+0000 7f53c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:40.218 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 589989919 0 0) 0x7f53cc139050 con 0x7f53cc0b4d60 2024-03-25T18:23:40.226 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 >> v1:172.21.15.87:6790/0 conn(0x7f53cc0b1730 legacy=0x7f53cc13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 >> v1:172.21.15.87:6789/0 conn(0x7f53cc14be90 legacy=0x7f53cc1342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 1 monclient: found mon.b 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc13a610 con 0x7f53cc0b4d60 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 10 monclient: _renew_subs 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc13a8f0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:40.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 10 monclient: _renew_subs 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc139730 con 0x7f53cc0b4d60 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.225+0000 7f53d261e640 1 librados: init done 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.226+0000 7f53d261e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.226+0000 7f53d261e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.226+0000 7f53d261e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:40.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.226+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc1391e0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c0003610 con 0x7f53cc0b4d60 2024-03-25T18:23:40.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:40.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f53c00038c0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.507 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53c8ff9640 10 monclient: dump: 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:40.508 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.506+0000 7f53ca7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:40.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.507+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f53c0003ae0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.508+0000 7f53cbfff640 10 monclient: get_auth_request con 0x7f53b806dfb0 auth_method 0 2024-03-25T18:23:40.509 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.508+0000 7f53c8ff9640 10 client.4633.objecter ms_handle_connect 0x7f53b806dfb0 2024-03-25T18:23:40.513 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.512+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f53c0053410 con 0x7f53cc0b4d60 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.512+0000 7f53c8ff9640 10 client.4633.objecter ms_dispatch 0x7f53cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.512+0000 7f53c8ff9640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.512+0000 7f53c8ff9640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.513+0000 7f53c8ff9640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.513+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53c004ef20 con 0x7f53cc0b4d60 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.513+0000 7f53c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:40.514 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.513+0000 7f53c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:40.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.743+0000 7f53d261e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:40.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.743+0000 7f53d261e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:40.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.743+0000 7f53d261e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:40.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.743+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53cc0627b0 con 0x7f53cc0b4d60 2024-03-25T18:23:40.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.789+0000 7f53c8ff9640 1 -- 172.21.15.87:0/1857600306 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f53c0050e50 con 0x7f53cc0b4d60 2024-03-25T18:23:40.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.789+0000 7f53c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:40.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.789+0000 7f53c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:40.791 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:40.791 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:40.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.791+0000 7f53d261e640 10 librados: watch_flush enter 2024-03-25T18:23:40.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 10 librados: watch_flush exit 2024-03-25T18:23:40.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 10 client.4633.objecter successfully canceled tick 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 >> v1:172.21.15.171:6800/3417775508 conn(0x7f53b806dfb0 legacy=0x7f53b8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 10 monclient: shutdown 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 >> v1:172.21.15.171:6789/0 conn(0x7f53cc0b4d60 legacy=0x7f53cc133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 shutdown_connections 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 >> 172.21.15.87:0/1857600306 conn(0x7f53cc0617a0 msgr2=0x7f53cc0b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 shutdown_connections 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 -- 172.21.15.87:0/1857600306 wait complete. 2024-03-25T18:23:40.794 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:40.792+0000 7f53d261e640 1 librados: shutdown 2024-03-25T18:23:40.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:47.611 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.726+0000 7f694c025640 10 monclient: get_monmap_and_config 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.726+0000 7f694c025640 10 monclient: build_initial_monmap 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.726+0000 7f694c025640 10 monclient: monmap: 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:47.726983+0000 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:47.726983+0000 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:47.727 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.728 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:47.728 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:47.728 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:47.728 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.726+0000 7f694c025640 1 Processor -- start 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 1 -- start start 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient: init 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient(hunting): picked mon.noname-b con 0x7f69440705b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient(hunting): picked mon.noname-a con 0x7f69441402d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 10 monclient(hunting): picked mon.noname-c con 0x7f6944149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440718c0 con 0x7f69441402d0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440714c0 con 0x7f6944149ae0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.727+0000 7f694c025640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944072b50 con 0x7f69440705b0 2024-03-25T18:23:47.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.728+0000 7f694c025640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:47.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.728+0000 7f694c025640 10 monclient(hunting): authenticate will time out at 8343.022849s 2024-03-25T18:23:47.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.731+0000 7f6949dc1640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f69440705b0 0x7f694413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:36082/0 (socket says 172.21.15.87:36082) 2024-03-25T18:23:47.731 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.731+0000 7f6949dc1640 1 -- 172.21.15.87:0/3039484626 learned_addr learned my addr 172.21.15.87:0/3039484626 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:47.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148174124 0 0) 0x7f6944072b50 con 0x7f69440705b0 2024-03-25T18:23:47.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.736 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:23:47.737 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.735+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6930003520 con 0x7f69440705b0 2024-03-25T18:23:47.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500317519 0 0) 0x7f69440714c0 con 0x7f6944149ae0 2024-03-25T18:23:47.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:23:47.746 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.744+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944072b50 con 0x7f6944149ae0 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321262493 0 0) 0x7f69440718c0 con 0x7f69441402d0 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:23:47.752 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.751+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69440714c0 con 0x7f69441402d0 2024-03-25T18:23:47.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751725892 0 0) 0x7f6930003520 con 0x7f69440705b0 2024-03-25T18:23:47.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 >> v1:172.21.15.171:6789/0 conn(0x7f6944149ae0 legacy=0x7f694414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:47.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 >> v1:172.21.15.87:6789/0 conn(0x7f69441402d0 legacy=0x7f6944140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:47.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: _finish_hunting 0 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 monclient: found mon.noname-b 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694413dd10 con 0x7f69440705b0 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: _finish_auth 0 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: _check_auth_tickets 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6940001690 con 0x7f69440705b0 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f6948dbf640 10 monclient: dump: 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.755 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f694c025640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:23:47.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.753+0000 7f694c025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69400017f0 con 0x7f69440705b0 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 1 -- 172.21.15.87:0/3039484626 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6940001c00 con 0x7f69440705b0 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f6948dbf640 10 monclient: dump: 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.770 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:47.771 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:47.771 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:47.771 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f694c025640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:47.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.768+0000 7f694c025640 4 set_mon_vals no callback set 2024-03-25T18:23:47.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.769+0000 7f694c025640 10 monclient: shutdown 2024-03-25T18:23:47.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.769+0000 7f694c025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:47.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.769+0000 7f694c025640 1 -- 172.21.15.87:0/3039484626 >> v1:172.21.15.87:6790/0 conn(0x7f69440705b0 legacy=0x7f694413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:47.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.769+0000 7f694c025640 1 -- 172.21.15.87:0/3039484626 shutdown_connections 2024-03-25T18:23:47.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.769+0000 7f694c025640 1 -- 172.21.15.87:0/3039484626 >> 172.21.15.87:0/3039484626 conn(0x7f69440b5d10 msgr2=0x7f69440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.770+0000 7f694c025640 1 -- 172.21.15.87:0/3039484626 shutdown_connections 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.770+0000 7f694c025640 1 -- 172.21.15.87:0/3039484626 wait complete. 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.770+0000 7f694c025640 10 monclient: build_initial_monmap 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.770+0000 7f694c025640 10 monclient: monmap: 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 librados: starting msgr at 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 librados: starting objecter 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 Processor -- start 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 -- start start 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 librados: setting wanted keys 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 1 librados: calling monclient init 2024-03-25T18:23:47.774 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.771+0000 7f694c025640 10 monclient: init 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient(hunting): picked mon.noname-c con 0x7f69440705b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient(hunting): picked mon.noname-a con 0x7f69441402d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient(hunting): picked mon.noname-b con 0x7f6944149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694406bc90 con 0x7f69441402d0 2024-03-25T18:23:47.775 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944139d50 con 0x7f6944149ae0 2024-03-25T18:23:47.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69441417a0 con 0x7f69440705b0 2024-03-25T18:23:47.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:47.776 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.772+0000 7f694c025640 10 monclient(hunting): authenticate will time out at 8343.067273s 2024-03-25T18:23:47.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.784+0000 7f6949dc1640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f69440705b0 0x7f6944139640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:36096/0 (socket says 172.21.15.87:36096) 2024-03-25T18:23:47.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.784+0000 7f6949dc1640 1 -- 172.21.15.87:0/1196375074 learned_addr learned my addr 172.21.15.87:0/1196375074 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:47.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.788+0000 7f693a7fc640 10 client.?.objecter ms_handle_connect 0x7f6944149ae0 2024-03-25T18:23:47.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.788+0000 7f693a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:47.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.789+0000 7f693a7fc640 10 client.?.objecter ms_handle_connect 0x7f69441402d0 2024-03-25T18:23:47.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.789+0000 7f693a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:47.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.789+0000 7f693a7fc640 10 client.?.objecter ms_handle_connect 0x7f69440705b0 2024-03-25T18:23:47.790 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.789+0000 7f693a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:47.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078128500 0 0) 0x7f6944139d50 con 0x7f6944149ae0 2024-03-25T18:23:47.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.797 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:23:47.798 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.796+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6928004eb0 con 0x7f6944149ae0 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604400644 0 0) 0x7f69441417a0 con 0x7f69440705b0 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:23:47.800 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.799+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944139d50 con 0x7f69440705b0 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881159698 0 0) 0x7f694406bc90 con 0x7f69441402d0 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:23:47.802 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.801+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69441417a0 con 0x7f69441402d0 2024-03-25T18:23:47.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2662819817 0 0) 0x7f6944139d50 con 0x7f69440705b0 2024-03-25T18:23:47.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694406bc90 con 0x7f69440705b0 2024-03-25T18:23:47.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f6940001b80 con 0x7f69440705b0 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.814 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.812+0000 7f693a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:47.820 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1050745051 0 0) 0x7f69441417a0 con 0x7f69441402d0 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6944139d50 con 0x7f69441402d0 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f69340018b0 con 0x7f69441402d0 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.819+0000 7f693a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:47.821 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814062522 0 0) 0x7f6928004eb0 con 0x7f6944149ae0 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69441417a0 con 0x7f6944149ae0 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f693c001af0 con 0x7f6944149ae0 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.820+0000 7f693a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:47.822 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:47.823 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1851810886 0 0) 0x7f69441417a0 con 0x7f6944149ae0 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 >> v1:172.21.15.87:6790/0 conn(0x7f69440705b0 legacy=0x7f6944139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 >> v1:172.21.15.87:6789/0 conn(0x7f69441402d0 legacy=0x7f6944133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 1 monclient: found mon.b 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694406bee0 con 0x7f6944149ae0 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:47.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f693a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 10 monclient: _renew_subs 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694413ff00 con 0x7f6944149ae0 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 10 monclient: _renew_subs 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f694406cfa0 con 0x7f6944149ae0 2024-03-25T18:23:47.837 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.834+0000 7f694c025640 1 librados: init done 2024-03-25T18:23:47.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.835+0000 7f694c025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:47.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.835+0000 7f694c025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:47.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.835+0000 7f694c025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:47.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:47.835+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69440627b0 con 0x7f6944149ae0 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f693c002000 con 0x7f6944149ae0 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f693c003b10 con 0x7f6944149ae0 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693a7fc640 10 monclient: dump: 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:48.123 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:48.124 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:48.124 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:48.124 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:48.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.121+0000 7f693bfff640 4 set_mon_vals no callback set 2024-03-25T18:23:48.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.122+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f693c019ae0 con 0x7f6944149ae0 2024-03-25T18:23:48.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.123+0000 7f6949dc1640 10 monclient: get_auth_request con 0x7f692806e0c0 auth_method 0 2024-03-25T18:23:48.125 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.123+0000 7f693a7fc640 10 client.4639.objecter ms_handle_connect 0x7f692806e0c0 2024-03-25T18:23:48.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.128+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f693c053790 con 0x7f6944149ae0 2024-03-25T18:23:48.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.128+0000 7f693a7fc640 10 client.4639.objecter ms_dispatch 0x7f6944000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:48.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.128+0000 7f693a7fc640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:48.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.128+0000 7f693a7fc640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:48.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.128+0000 7f693a7fc640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:23:48.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.129+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f693c04f0d0 con 0x7f6944149ae0 2024-03-25T18:23:48.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.129+0000 7f693a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:48.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.129+0000 7f693a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:48.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.379+0000 7f694c025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:48.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.379+0000 7f694c025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:48.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.379+0000 7f694c025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:23:48.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.379+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69440b4c70 con 0x7f6944149ae0 2024-03-25T18:23:48.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.423+0000 7f693a7fc640 1 -- 172.21.15.87:0/1196375074 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f693c0511d0 con 0x7f6944149ae0 2024-03-25T18:23:48.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.423+0000 7f693a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:48.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.423+0000 7f693a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:48.425 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:48.425 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:48.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.425+0000 7f694c025640 10 librados: watch_flush enter 2024-03-25T18:23:48.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.425+0000 7f694c025640 10 librados: watch_flush exit 2024-03-25T18:23:48.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.425+0000 7f694c025640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:23:48.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.425+0000 7f694c025640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 >> v1:172.21.15.171:6800/3417775508 conn(0x7f692806e0c0 legacy=0x7f6928070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 10 monclient: shutdown 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 >> v1:172.21.15.171:6789/0 conn(0x7f6944149ae0 legacy=0x7f69441e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 shutdown_connections 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 >> 172.21.15.87:0/1196375074 conn(0x7f69440b5d10 msgr2=0x7f69440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 shutdown_connections 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 -- 172.21.15.87:0/1196375074 wait complete. 2024-03-25T18:23:48.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:48.426+0000 7f694c025640 1 librados: shutdown 2024-03-25T18:23:48.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:53.173 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:40.886 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.181 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:40.895 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.223 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:09:40.937 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.232 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:40.911 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.239 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:40.952 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.244 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:40.957 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.244 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:40.922 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.269 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:09:40.982 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.276 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:09:40.954 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:55.245 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:55.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.344+0000 7f4af6afb640 10 monclient: get_monmap_and_config 2024-03-25T18:23:55.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.344+0000 7f4af6afb640 10 monclient: build_initial_monmap 2024-03-25T18:23:55.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.344+0000 7f4af6afb640 10 monclient: monmap: 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:23:55.346698+0000 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:23:55.346698+0000 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:23:55.347 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.345+0000 7f4af6afb640 1 Processor -- start 2024-03-25T18:23:55.348 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.345+0000 7f4af6afb640 1 -- start start 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.345+0000 7f4af6afb640 10 monclient: init 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-b con 0x7f4af00b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-a con 0x7f4af00b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-c con 0x7f4af014be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af00af940 con 0x7f4af00b1730 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af00b0a70 con 0x7f4af014be90 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af00b0d50 con 0x7f4af00b4d60 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:55.349 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.346+0000 7f4af6afb640 10 monclient(hunting): authenticate will time out at 8350.642466s 2024-03-25T18:23:55.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.356+0000 7f4af5098640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4af014be90 0x7f4af014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:50316/0 (socket says 172.21.15.87:50316) 2024-03-25T18:23:55.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.356+0000 7f4af5098640 1 -- 172.21.15.87:0/4144511056 learned_addr learned my addr 172.21.15.87:0/4144511056 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:55.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864247682 0 0) 0x7f4af00b0a70 con 0x7f4af014be90 2024-03-25T18:23:55.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:55.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:55.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:55.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:23:55.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.365+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae0003520 con 0x7f4af014be90 2024-03-25T18:23:55.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1331266536 0 0) 0x7f4ae0003520 con 0x7f4af014be90 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 >> v1:172.21.15.87:6790/0 conn(0x7f4af00b4d60 legacy=0x7f4af013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 >> v1:172.21.15.87:6789/0 conn(0x7f4af00b1730 legacy=0x7f4af00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 monclient: found mon.noname-c 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00b1030 con 0x7f4af014be90 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4af6afb640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4af6afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ae8001a10 con 0x7f4af014be90 2024-03-25T18:23:55.377 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4aef7fe640 10 monclient: dump: 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.378 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.374+0000 7f4af6afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:55.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae8001be0 con 0x7f4af014be90 2024-03-25T18:23:55.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:55.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 1 -- 172.21.15.87:0/4144511056 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ae8002030 con 0x7f4af014be90 2024-03-25T18:23:55.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4af6afb640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:55.402 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4af6afb640 4 set_mon_vals no callback set 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.400+0000 7f4aef7fe640 10 monclient: dump: 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:55.403 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 10 monclient: shutdown 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 1 -- 172.21.15.87:0/4144511056 >> v1:172.21.15.171:6789/0 conn(0x7f4af014be90 legacy=0x7f4af014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 1 -- 172.21.15.87:0/4144511056 shutdown_connections 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 1 -- 172.21.15.87:0/4144511056 >> 172.21.15.87:0/4144511056 conn(0x7f4af00617a0 msgr2=0x7f4af0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 1 -- 172.21.15.87:0/4144511056 shutdown_connections 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.401+0000 7f4af6afb640 1 -- 172.21.15.87:0/4144511056 wait complete. 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.402+0000 7f4af6afb640 10 monclient: build_initial_monmap 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.402+0000 7f4af6afb640 10 monclient: monmap: 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:23:55.405 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.402+0000 7f4af6afb640 1 librados: starting msgr at 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.402+0000 7f4af6afb640 1 librados: starting objecter 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 1 Processor -- start 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 1 -- start start 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 1 librados: setting wanted keys 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 1 librados: calling monclient init 2024-03-25T18:23:55.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 10 monclient: init 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.403+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-c con 0x7f4af00b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-b con 0x7f4af00b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 10 monclient(hunting): picked mon.noname-a con 0x7f4af014be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0137fe0 con 0x7f4af014be90 2024-03-25T18:23:55.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af01384c0 con 0x7f4af00b4d60 2024-03-25T18:23:55.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0137570 con 0x7f4af00b1730 2024-03-25T18:23:55.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:55.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.404+0000 7f4af6afb640 10 monclient(hunting): authenticate will time out at 8350.699973s 2024-03-25T18:23:55.411 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.410+0000 7f4af4897640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4af00b1730 0x7f4af01426f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:40614/0 (socket says 172.21.15.87:40614) 2024-03-25T18:23:55.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.410+0000 7f4af4897640 1 -- 172.21.15.87:0/281342803 learned_addr learned my addr 172.21.15.87:0/281342803 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:23:55.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.414+0000 7f4aed7fa640 10 client.?.objecter ms_handle_connect 0x7f4af00b1730 2024-03-25T18:23:55.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.414+0000 7f4aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:55.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.421+0000 7f4aed7fa640 10 client.?.objecter ms_handle_connect 0x7f4af014be90 2024-03-25T18:23:55.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.421+0000 7f4aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039145713 0 0) 0x7f4af0137570 con 0x7f4af00b1730 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:23:55.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.422+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4004eb0 con 0x7f4af00b1730 2024-03-25T18:23:55.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.429+0000 7f4aed7fa640 10 client.?.objecter ms_handle_connect 0x7f4af00b4d60 2024-03-25T18:23:55.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.429+0000 7f4aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084652051 0 0) 0x7f4af0137fe0 con 0x7f4af014be90 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:23:55.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.431+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af0137570 con 0x7f4af014be90 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715444894 0 0) 0x7f4ad4004eb0 con 0x7f4af00b1730 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af0137fe0 con 0x7f4af00b1730 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ae4001960 con 0x7f4af00b1730 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 10 monclient(hunting): dump: 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:55.437 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.435+0000 7f4aed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557061000 0 0) 0x7f4af01384c0 con 0x7f4af00b4d60 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:55.438 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:23:55.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.436+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4004eb0 con 0x7f4af00b4d60 2024-03-25T18:23:55.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3043456177 0 0) 0x7f4af0137fe0 con 0x7f4af00b1730 2024-03-25T18:23:55.442 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 >> v1:172.21.15.171:6789/0 conn(0x7f4af00b4d60 legacy=0x7f4af01319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 >> v1:172.21.15.87:6789/0 conn(0x7f4af014be90 legacy=0x7f4af01320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 1 monclient: found mon.c 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0140fb0 con 0x7f4af00b1730 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 10 monclient: _finish_auth 0 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4aed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 5 monclient: authenticate success, global_id 4646 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 10 monclient: _renew_subs 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0141280 con 0x7f4af00b1730 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 10 monclient: _renew_subs 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:55.443 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af01406f0 con 0x7f4af00b1730 2024-03-25T18:23:55.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.440+0000 7f4af6afb640 1 librados: init done 2024-03-25T18:23:55.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.441+0000 7f4af6afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:55.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.441+0000 7f4af6afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:55.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.441+0000 7f4af6afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:55.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.441+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af00627b0 con 0x7f4af00b1730 2024-03-25T18:23:55.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae40035f0 con 0x7f4af00b1730 2024-03-25T18:23:55.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:55.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ae4003940 con 0x7f4af00b1730 2024-03-25T18:23:55.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:55.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aed7fa640 10 monclient: dump: 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:23:55.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.756+0000 7f4aeeffd640 4 set_mon_vals no callback set 2024-03-25T18:23:55.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.757+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4ae4003b60 con 0x7f4af00b1730 2024-03-25T18:23:55.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.757+0000 7f4aeffff640 10 monclient: get_auth_request con 0x7f4ad406de00 auth_method 0 2024-03-25T18:23:55.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.758+0000 7f4aed7fa640 10 client.4646.objecter ms_handle_connect 0x7f4ad406de00 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4ae40547a0 con 0x7f4af00b1730 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 10 client.4646.objecter ms_dispatch 0x7f4af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ae40502b0 con 0x7f4af00b1730 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:55.766 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:55.764+0000 7f4aed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:56.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.027+0000 7f4af6afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:56.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.027+0000 7f4af6afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:56.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.027+0000 7f4af6afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:23:56.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.027+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af00b5f40 con 0x7f4af00b1730 2024-03-25T18:23:56.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.060+0000 7f4aed7fa640 1 -- 172.21.15.87:0/281342803 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4ae40521e0 con 0x7f4af00b1730 2024-03-25T18:23:56.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.060+0000 7f4aed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:56.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.060+0000 7f4aed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:56.062 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:23:56.062 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:56.064 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 10 librados: watch_flush enter 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 10 librados: watch_flush exit 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 10 client.4646.objecter successfully canceled tick 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4ad406de00 legacy=0x7f4ad40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 10 monclient: shutdown 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 >> v1:172.21.15.87:6790/0 conn(0x7f4af00b1730 legacy=0x7f4af01426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 shutdown_connections 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.062+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 >> 172.21.15.87:0/281342803 conn(0x7f4af00617a0 msgr2=0x7f4af00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.063+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 shutdown_connections 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.063+0000 7f4af6afb640 1 -- 172.21.15.87:0/281342803 wait complete. 2024-03-25T18:23:56.065 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:23:56.063+0000 7f4af6afb640 1 librados: shutdown 2024-03-25T18:23:56.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:02.884 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.983+0000 7f75ee956640 10 monclient: get_monmap_and_config 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.983+0000 7f75ee956640 10 monclient: build_initial_monmap 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.983+0000 7f75ee956640 10 monclient: monmap: 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:02.983960+0000 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:02.983960+0000 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:02.984 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:02.985 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:02.985 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 1 Processor -- start 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 1 -- start start 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient: init 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-b con 0x7f75e80b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:02.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-a con 0x7f75e80b1730 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-c con 0x7f75e814be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80af940 con 0x7f75e80b1730 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80b0a70 con 0x7f75e814be90 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80b0d50 con 0x7f75e80b4d60 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:02.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.984+0000 7f75ee956640 10 monclient(hunting): authenticate will time out at 8358.279832s 2024-03-25T18:24:02.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.994+0000 7f75e7fff640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f75e80b4d60 0x7f75e813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:59086/0 (socket says 172.21.15.87:59086) 2024-03-25T18:24:02.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:02.994+0000 7f75e7fff640 1 -- 172.21.15.87:0/2247429272 learned_addr learned my addr 172.21.15.87:0/2247429272 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859797520 0 0) 0x7f75e80b0a70 con 0x7f75e814be90 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:24:03.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.016+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d0003520 con 0x7f75e814be90 2024-03-25T18:24:03.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849429821 0 0) 0x7f75e80b0d50 con 0x7f75e80b4d60 2024-03-25T18:24:03.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:03.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:03.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:03.022 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:24:03.023 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.021+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e80b0a70 con 0x7f75e80b4d60 2024-03-25T18:24:03.024 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659971723 0 0) 0x7f75d0003520 con 0x7f75e814be90 2024-03-25T18:24:03.024 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 >> v1:172.21.15.87:6790/0 conn(0x7f75e80b4d60 legacy=0x7f75e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.024 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 >> v1:172.21.15.87:6789/0 conn(0x7f75e80b1730 legacy=0x7f75e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e80b1030 con 0x7f75e814be90 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75e0001a10 con 0x7f75e814be90 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75e6ffd640 10 monclient: dump: 2024-03-25T18:24:03.025 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75ee956640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:24:03.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.023+0000 7f75ee956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:03.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e0001b70 con 0x7f75e814be90 2024-03-25T18:24:03.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:03.033 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 1 -- 172.21.15.87:0/2247429272 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75e0001f80 con 0x7f75e814be90 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75e6ffd640 10 monclient: dump: 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.034 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75ee956640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:03.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.032+0000 7f75ee956640 4 set_mon_vals no callback set 2024-03-25T18:24:03.036 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 10 monclient: shutdown 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 1 -- 172.21.15.87:0/2247429272 >> v1:172.21.15.171:6789/0 conn(0x7f75e814be90 legacy=0x7f75e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 1 -- 172.21.15.87:0/2247429272 shutdown_connections 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 1 -- 172.21.15.87:0/2247429272 >> 172.21.15.87:0/2247429272 conn(0x7f75e80617a0 msgr2=0x7f75e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 1 -- 172.21.15.87:0/2247429272 shutdown_connections 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.033+0000 7f75ee956640 1 -- 172.21.15.87:0/2247429272 wait complete. 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.034+0000 7f75ee956640 10 monclient: build_initial_monmap 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.034+0000 7f75ee956640 10 monclient: monmap: 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:03.037 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.034+0000 7f75ee956640 1 librados: starting msgr at 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.034+0000 7f75ee956640 1 librados: starting objecter 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.034+0000 7f75ee956640 1 Processor -- start 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 -- start start 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 librados: setting wanted keys 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 librados: calling monclient init 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient: init 2024-03-25T18:24:03.038 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-b con 0x7f75e80b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-c con 0x7f75e80b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient(hunting): picked mon.noname-a con 0x7f75e814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8140380 con 0x7f75e814be90 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8140660 con 0x7f75e80b1730 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8141e30 con 0x7f75e80b4d60 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:03.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.035+0000 7f75ee956640 10 monclient(hunting): authenticate will time out at 8358.330723s 2024-03-25T18:24:03.048 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.047+0000 7f75e7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f75e80b1730 0x7f75e8138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52068/0 (socket says 172.21.15.87:52068) 2024-03-25T18:24:03.049 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.048+0000 7f75e7fff640 1 -- 172.21.15.87:0/1913132444 learned_addr learned my addr 172.21.15.87:0/1913132444 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:03.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.050+0000 7f75e4ff9640 10 client.?.objecter ms_handle_connect 0x7f75e80b1730 2024-03-25T18:24:03.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.050+0000 7f75e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:03.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.054+0000 7f75e4ff9640 10 client.?.objecter ms_handle_connect 0x7f75e814be90 2024-03-25T18:24:03.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.054+0000 7f75e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:03.056 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.055+0000 7f75e4ff9640 10 client.?.objecter ms_handle_connect 0x7f75e80b4d60 2024-03-25T18:24:03.057 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.055+0000 7f75e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.067+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321590357 0 0) 0x7f75e8140380 con 0x7f75e814be90 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.067+0000 7f75e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.067+0000 7f75e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.067+0000 7f75e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4004eb0 con 0x7f75e814be90 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049037634 0 0) 0x7f75e8140660 con 0x7f75e80b1730 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:03.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:24:03.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.068+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e8140380 con 0x7f75e80b1730 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209117665 0 0) 0x7f75e8141e30 con 0x7f75e80b4d60 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:24:03.072 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.071+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e8140660 con 0x7f75e80b4d60 2024-03-25T18:24:03.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638904833 0 0) 0x7f75e8140380 con 0x7f75e80b1730 2024-03-25T18:24:03.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e8141e30 con 0x7f75e80b1730 2024-03-25T18:24:03.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75d8001930 con 0x7f75e80b1730 2024-03-25T18:24:03.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.087+0000 7f75e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753973635 0 0) 0x7f75e8140660 con 0x7f75e80b4d60 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e8140380 con 0x7f75e80b4d60 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75dc001860 con 0x7f75e80b4d60 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:03.098 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.097+0000 7f75e4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.099 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.106 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539389216 0 0) 0x7f75d4004eb0 con 0x7f75e814be90 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e8140660 con 0x7f75e814be90 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75e0001ec0 con 0x7f75e814be90 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.105+0000 7f75e4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.107 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.108 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4197742794 0 0) 0x7f75e8141e30 con 0x7f75e80b1730 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 >> v1:172.21.15.87:6790/0 conn(0x7f75e80b4d60 legacy=0x7f75e8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 >> v1:172.21.15.87:6789/0 conn(0x7f75e814be90 legacy=0x7f75e81e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 1 monclient: found mon.b 2024-03-25T18:24:03.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8142110 con 0x7f75e80b1730 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 10 monclient: _renew_subs 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e8140e60 con 0x7f75e80b1730 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 10 monclient: _renew_subs 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e8142c40 con 0x7f75e80b1730 2024-03-25T18:24:03.115 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.113+0000 7f75ee956640 1 librados: init done 2024-03-25T18:24:03.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.114+0000 7f75ee956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:03.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.114+0000 7f75ee956640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:03.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.114+0000 7f75ee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.116 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.114+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e80627b0 con 0x7f75e80b1730 2024-03-25T18:24:03.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.327+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d8001d40 con 0x7f75e80b1730 2024-03-25T18:24:03.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.327+0000 7f75e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:03.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.327+0000 7f75e67fc640 4 set_mon_vals no callback set 2024-03-25T18:24:03.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.385+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f75d8004980 con 0x7f75e80b1730 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.385+0000 7f75e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.385+0000 7f75e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.385+0000 7f75e4ff9640 10 monclient: dump: 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:03.387 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:03.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.386+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f75d8004b60 con 0x7f75e80b1730 2024-03-25T18:24:03.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.387+0000 7f75e77fe640 10 monclient: get_auth_request con 0x7f75d406e0f0 auth_method 0 2024-03-25T18:24:03.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.387+0000 7f75e4ff9640 10 client.4651.objecter ms_handle_connect 0x7f75d406e0f0 2024-03-25T18:24:03.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.389+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f75d80547f0 con 0x7f75e80b1730 2024-03-25T18:24:03.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.389+0000 7f75e4ff9640 10 client.4651.objecter ms_dispatch 0x7f75e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:03.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.389+0000 7f75e4ff9640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:03.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.389+0000 7f75e4ff9640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:03.391 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.389+0000 7f75e4ff9640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:24:03.394 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.393+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75d8050130 con 0x7f75e80b1730 2024-03-25T18:24:03.394 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.393+0000 7f75e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:03.394 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.393+0000 7f75e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:03.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.654+0000 7f75ee956640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:03.655 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.654+0000 7f75ee956640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:03.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.654+0000 7f75ee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:03.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.654+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75e80b5cf0 con 0x7f75e80b1730 2024-03-25T18:24:03.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.685+0000 7f75e4ff9640 1 -- 172.21.15.87:0/1913132444 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f75d8052230 con 0x7f75e80b1730 2024-03-25T18:24:03.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.685+0000 7f75e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:03.686 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.685+0000 7f75e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:03.687 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:03.687 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.687+0000 7f75ee956640 10 librados: watch_flush enter 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 10 librados: watch_flush exit 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 >> v1:172.21.15.171:6800/3417775508 conn(0x7f75d406e0f0 legacy=0x7f75d40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 10 monclient: shutdown 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 >> v1:172.21.15.171:6789/0 conn(0x7f75e80b1730 legacy=0x7f75e8138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 shutdown_connections 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.688+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 >> 172.21.15.87:0/1913132444 conn(0x7f75e80617a0 msgr2=0x7f75e80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.689+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 shutdown_connections 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.689+0000 7f75ee956640 1 -- 172.21.15.87:0/1913132444 wait complete. 2024-03-25T18:24:03.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:03.689+0000 7f75ee956640 1 librados: shutdown 2024-03-25T18:24:03.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:10.507 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.609+0000 7f24b241e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.609+0000 7f24b241e640 10 monclient: build_initial_monmap 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.609+0000 7f24b241e640 10 monclient: monmap: 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:10.610056+0000 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:10.610056+0000 2024-03-25T18:24:10.610 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:10.611 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:10.611 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:10.611 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:10.611 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:10.611 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 1 Processor -- start 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 1 -- start start 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient: init 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac0a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-c con 0x7f24ac13f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:10.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac1430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:10.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac142520 con 0x7f24ac1430c0 2024-03-25T18:24:10.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0b28a0 con 0x7f24ac13f710 2024-03-25T18:24:10.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0b2a30 con 0x7f24ac0a98f0 2024-03-25T18:24:10.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:10.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.610+0000 7f24b241e640 10 monclient(hunting): authenticate will time out at 8365.905734s 2024-03-25T18:24:10.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.614+0000 7f24ab7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f24ac13f710 0x7f24ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:38440/0 (socket says 172.21.15.87:38440) 2024-03-25T18:24:10.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.614+0000 7f24ab7fe640 1 -- 172.21.15.87:0/1268034387 learned_addr learned my addr 172.21.15.87:0/1268034387 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799594926 0 0) 0x7f24ac142520 con 0x7f24ac1430c0 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:24:10.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.701+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2494003520 con 0x7f24ac1430c0 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515781343 0 0) 0x7f2494003520 con 0x7f24ac1430c0 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 >> v1:172.21.15.87:6790/0 conn(0x7f24ac0a98f0 legacy=0x7f24ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 >> v1:172.21.15.171:6789/0 conn(0x7f24ac13f710 legacy=0x7f24ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 monclient: found mon.noname-a 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac0b44c0 con 0x7f24ac1430c0 2024-03-25T18:24:10.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f24a4001a10 con 0x7f24ac1430c0 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24b241e640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24aaffd640 10 monclient: dump: 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:10.729 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:10.730 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.727+0000 7f24b241e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:10.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a4001b70 con 0x7f24ac1430c0 2024-03-25T18:24:10.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:10.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 1 -- 172.21.15.87:0/1268034387 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f24a4001f80 con 0x7f24ac1430c0 2024-03-25T18:24:10.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24b241e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:10.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24b241e640 4 set_mon_vals no callback set 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.743+0000 7f24aaffd640 10 monclient: dump: 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:10.745 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 10 monclient: shutdown 2024-03-25T18:24:10.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:10.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 1 -- 172.21.15.87:0/1268034387 >> v1:172.21.15.87:6789/0 conn(0x7f24ac1430c0 legacy=0x7f24ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.747 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 1 -- 172.21.15.87:0/1268034387 shutdown_connections 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 1 -- 172.21.15.87:0/1268034387 >> 172.21.15.87:0/1268034387 conn(0x7f24ac0b7ea0 msgr2=0x7f24ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 1 -- 172.21.15.87:0/1268034387 shutdown_connections 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.744+0000 7f24b241e640 1 -- 172.21.15.87:0/1268034387 wait complete. 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 10 monclient: build_initial_monmap 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 10 monclient: monmap: 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:10.748 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 librados: starting msgr at 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 librados: starting objecter 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 Processor -- start 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 -- start start 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 librados: setting wanted keys 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 1 librados: calling monclient init 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.745+0000 7f24b241e640 10 monclient: init 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-c con 0x7f24ac0a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac13f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac1430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac138b60 con 0x7f24ac13f710 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac137e40 con 0x7f24ac1430c0 2024-03-25T18:24:10.749 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac1d82f0 con 0x7f24ac0a98f0 2024-03-25T18:24:10.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:10.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.746+0000 7f24b241e640 10 monclient(hunting): authenticate will time out at 8366.041301s 2024-03-25T18:24:10.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.756+0000 7f24ab7fe640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f24ac13f710 0x7f24ac1d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:45096/0 (socket says 172.21.15.87:45096) 2024-03-25T18:24:10.757 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.756+0000 7f24ab7fe640 1 -- 172.21.15.87:0/1547578226 learned_addr learned my addr 172.21.15.87:0/1547578226 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:10.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.757+0000 7f24a8ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac13f710 2024-03-25T18:24:10.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.757+0000 7f24a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:10.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398299093 0 0) 0x7f24ac138b60 con 0x7f24ac13f710 2024-03-25T18:24:10.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:10.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:10.769 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:10.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:24:10.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.768+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2498004eb0 con 0x7f24ac13f710 2024-03-25T18:24:10.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.769+0000 7f24a8ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac1430c0 2024-03-25T18:24:10.770 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.769+0000 7f24a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971977383 0 0) 0x7f24ac137e40 con 0x7f24ac1430c0 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:24:10.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.782+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac138b60 con 0x7f24ac1430c0 2024-03-25T18:24:10.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.790+0000 7f24a8ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac0a98f0 2024-03-25T18:24:10.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.790+0000 7f24a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.790+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250296834 0 0) 0x7f2498004eb0 con 0x7f24ac13f710 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ac137e40 con 0x7f24ac13f710 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f24a00018b0 con 0x7f24ac13f710 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:10.792 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.793 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.791+0000 7f24a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168688708 0 0) 0x7f24ac1d82f0 con 0x7f24ac0a98f0 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:24:10.810 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.809+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2498004eb0 con 0x7f24ac0a98f0 2024-03-25T18:24:10.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309752172 0 0) 0x7f24ac138b60 con 0x7f24ac1430c0 2024-03-25T18:24:10.812 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ac1d82f0 con 0x7f24ac1430c0 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f24a4001ec0 con 0x7f24ac1430c0 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.811+0000 7f24a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:10.813 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2961852862 0 0) 0x7f24ac137e40 con 0x7f24ac13f710 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 >> v1:172.21.15.87:6790/0 conn(0x7f24ac0a98f0 legacy=0x7f24ac137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 >> v1:172.21.15.171:6789/0 conn(0x7f24ac1430c0 legacy=0x7f24ac133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 1 monclient: found mon.a 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac134e70 con 0x7f24ac13f710 2024-03-25T18:24:10.826 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 10 monclient: _renew_subs 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac1350c0 con 0x7f24ac13f710 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 10 client.4650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 10 monclient: _renew_subs 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac134800 con 0x7f24ac13f710 2024-03-25T18:24:10.827 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.824+0000 7f24b241e640 1 librados: init done 2024-03-25T18:24:10.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.825+0000 7f24b241e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:10.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.825+0000 7f24b241e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:10.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.825+0000 7f24b241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:10.828 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:10.825+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ac134ab0 con 0x7f24ac13f710 2024-03-25T18:24:11.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a00047e0 con 0x7f24ac13f710 2024-03-25T18:24:11.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:11.121 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f24a0004c30 con 0x7f24ac13f710 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24a8ff9640 10 monclient: dump: 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:11.122 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:11.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.120+0000 7f24aa7fc640 4 set_mon_vals no callback set 2024-03-25T18:24:11.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.121+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f24a0004e50 con 0x7f24ac13f710 2024-03-25T18:24:11.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.121+0000 7f24abfff640 10 monclient: get_auth_request con 0x7f249806dfb0 auth_method 0 2024-03-25T18:24:11.123 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.122+0000 7f24a8ff9640 10 client.4650.objecter ms_handle_connect 0x7f249806dfb0 2024-03-25T18:24:11.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f24a00547c0 con 0x7f24ac13f710 2024-03-25T18:24:11.129 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 10 client.4650.objecter ms_dispatch 0x7f24ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 3 client.4650.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 3 client.4650.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 20 client.4650.objecter dump_active .. 0 homeless 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24a00502d0 con 0x7f24ac13f710 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:11.130 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.128+0000 7f24a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:11.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.389+0000 7f24b241e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:11.391 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.389+0000 7f24b241e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:11.391 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.389+0000 7f24b241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:11.391 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.389+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24ac0abb30 con 0x7f24ac13f710 2024-03-25T18:24:11.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.423+0000 7f24a8ff9640 1 -- 172.21.15.87:0/1547578226 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f24a0052200 con 0x7f24ac13f710 2024-03-25T18:24:11.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.423+0000 7f24a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:11.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.423+0000 7f24a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:11.425 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:11.425 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:11.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 10 librados: watch_flush enter 2024-03-25T18:24:11.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 10 librados: watch_flush exit 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 20 client.4650.objecter shutdown clearing up homeless session... 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 10 client.4650.objecter successfully canceled tick 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 >> v1:172.21.15.171:6800/3417775508 conn(0x7f249806dfb0 legacy=0x7f2498070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 10 monclient: shutdown 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 >> v1:172.21.15.87:6789/0 conn(0x7f24ac13f710 legacy=0x7f24ac1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 shutdown_connections 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 >> 172.21.15.87:0/1547578226 conn(0x7f24ac0b7ea0 msgr2=0x7f24ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 shutdown_connections 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 -- 172.21.15.87:0/1547578226 wait complete. 2024-03-25T18:24:11.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:11.426+0000 7f24b241e640 1 librados: shutdown 2024-03-25T18:24:11.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:13.173 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:00.886 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.219 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:00.932 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.220 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:00.933 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.224 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:00.937 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.230 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:00.943 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.259 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:00.972 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.281 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:00.960 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.296 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:00.974 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.318 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:00.996 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:18.246 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:18.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.365+0000 7f9c9a15e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:18.366 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.365+0000 7f9c9a15e640 10 monclient: build_initial_monmap 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.365+0000 7f9c9a15e640 10 monclient: monmap: 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:18.366447+0000 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:18.366447+0000 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:18.367 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.366+0000 7f9c9a15e640 1 Processor -- start 2024-03-25T18:24:18.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.366+0000 7f9c9a15e640 1 -- start start 2024-03-25T18:24:18.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.366+0000 7f9c9a15e640 10 monclient: init 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-b con 0x7f9c940b10d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-c con 0x7f9c940b4360 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-a con 0x7f9c940a9470 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940a9ab0 con 0x7f9c940a9470 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940b2400 con 0x7f9c940b4360 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940aa2a0 con 0x7f9c940b10d0 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:18.369 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.367+0000 7f9c9a15e640 10 monclient(hunting): authenticate will time out at 8373.662195s 2024-03-25T18:24:18.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.378+0000 7f9c92ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9c940b4360 0x7f9c940b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:48188/0 (socket says 172.21.15.87:48188) 2024-03-25T18:24:18.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.378+0000 7f9c92ffd640 1 -- 172.21.15.87:0/1874683474 learned_addr learned my addr 172.21.15.87:0/1874683474 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.388+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849033743 0 0) 0x7f9c940b2400 con 0x7f9c940b4360 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.389+0000 7f9c927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.389+0000 7f9c927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.389+0000 7f9c927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.389+0000 7f9c927fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:24:18.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.389+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c7c003520 con 0x7f9c940b4360 2024-03-25T18:24:18.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757230488 0 0) 0x7f9c7c003520 con 0x7f9c940b4360 2024-03-25T18:24:18.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 >> v1:172.21.15.87:6790/0 conn(0x7f9c940b10d0 legacy=0x7f9c940b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 >> v1:172.21.15.87:6789/0 conn(0x7f9c940a9470 legacy=0x7f9c9414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.395 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c940b14b0 con 0x7f9c940b4360 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c9a15e640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:24:18.396 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.394+0000 7f9c9a15e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9c88001690 con 0x7f9c940b4360 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: dump: 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:18.421 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c88001920 con 0x7f9c940b4360 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 1 -- 172.21.15.87:0/1874683474 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9c88001c30 con 0x7f9c940b4360 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c927fc640 10 monclient: dump: 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.422 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c9a15e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:18.423 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.419+0000 7f9c9a15e640 4 set_mon_vals no callback set 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 10 monclient: shutdown 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1874683474 >> v1:172.21.15.171:6789/0 conn(0x7f9c940b4360 legacy=0x7f9c940b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1874683474 shutdown_connections 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1874683474 >> 172.21.15.87:0/1874683474 conn(0x7f9c940617a0 msgr2=0x7f9c94061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1874683474 shutdown_connections 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.420+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1874683474 wait complete. 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.421+0000 7f9c9a15e640 10 monclient: build_initial_monmap 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.421+0000 7f9c9a15e640 10 monclient: monmap: 2024-03-25T18:24:18.425 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 librados: starting msgr at 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 librados: starting objecter 2024-03-25T18:24:18.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 Processor -- start 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 -- start start 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 librados: setting wanted keys 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 1 librados: calling monclient init 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.422+0000 7f9c9a15e640 10 monclient: init 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-a con 0x7f9c940a9470 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-c con 0x7f9c940b10d0 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient(hunting): picked mon.noname-b con 0x7f9c940b4360 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940ab8a0 con 0x7f9c940a9470 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940abd30 con 0x7f9c940b4360 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c940aca00 con 0x7f9c940b10d0 2024-03-25T18:24:18.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:18.428 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.423+0000 7f9c9a15e640 10 monclient(hunting): authenticate will time out at 8373.718106s 2024-03-25T18:24:18.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.430+0000 7f9c93fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9c940b4360 0x7f9c941daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:48238/0 (socket says 172.21.15.87:48238) 2024-03-25T18:24:18.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.430+0000 7f9c93fff640 1 -- 172.21.15.87:0/1837873068 learned_addr learned my addr 172.21.15.87:0/1837873068 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:18.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.433+0000 7f9c7bfff640 10 client.?.objecter ms_handle_connect 0x7f9c940b4360 2024-03-25T18:24:18.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.433+0000 7f9c7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:18.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.435+0000 7f9c7bfff640 10 client.?.objecter ms_handle_connect 0x7f9c940a9470 2024-03-25T18:24:18.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.435+0000 7f9c7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:18.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.439+0000 7f9c7bfff640 10 client.?.objecter ms_handle_connect 0x7f9c940b10d0 2024-03-25T18:24:18.439 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.439+0000 7f9c7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.442+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842338244 0 0) 0x7f9c940ab8a0 con 0x7f9c940a9470 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.443+0000 7f9c7bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.443+0000 7f9c7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.443+0000 7f9c7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.443+0000 7f9c7bfff640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:24:18.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.443+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c80004eb0 con 0x7f9c940a9470 2024-03-25T18:24:18.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155135190 0 0) 0x7f9c80004eb0 con 0x7f9c940a9470 2024-03-25T18:24:18.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c940ab8a0 con 0x7f9c940a9470 2024-03-25T18:24:18.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9c840018c0 con 0x7f9c940a9470 2024-03-25T18:24:18.458 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.457+0000 7f9c7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 265184762 0 0) 0x7f9c940ab8a0 con 0x7f9c940a9470 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 >> v1:172.21.15.87:6790/0 conn(0x7f9c940b10d0 legacy=0x7f9c94139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 >> v1:172.21.15.171:6789/0 conn(0x7f9c940b4360 legacy=0x7f9c941daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 1 monclient: found mon.a 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.469+0000 7f9c7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:18.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c940acb90 con 0x7f9c940a9470 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c7bfff640 10 monclient: _finish_auth 0 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c7bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 10 monclient: _renew_subs 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c940b06b0 con 0x7f9c940a9470 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 10 monclient: _renew_subs 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9413f9e0 con 0x7f9c940a9470 2024-03-25T18:24:18.472 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.470+0000 7f9c9a15e640 1 librados: init done 2024-03-25T18:24:18.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.471+0000 7f9c9a15e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:18.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.471+0000 7f9c9a15e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:18.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.471+0000 7f9c9a15e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:18.473 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.471+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c940b0940 con 0x7f9c940a9470 2024-03-25T18:24:18.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.676+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c840035f0 con 0x7f9c940a9470 2024-03-25T18:24:18.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.676+0000 7f9c7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:18.677 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.676+0000 7f9c91ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:18.711 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.711+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f9c84003820 con 0x7f9c940a9470 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.711+0000 7f9c7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.711+0000 7f9c7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.711+0000 7f9c7bfff640 10 monclient: dump: 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:18.712 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:18.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.711+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f9c84003a00 con 0x7f9c940a9470 2024-03-25T18:24:18.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.712+0000 7f9c92ffd640 10 monclient: get_auth_request con 0x7f9c8006dc00 auth_method 0 2024-03-25T18:24:18.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.712+0000 7f9c7bfff640 10 client.4656.objecter ms_handle_connect 0x7f9c8006dc00 2024-03-25T18:24:18.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f9c84053520 con 0x7f9c940a9470 2024-03-25T18:24:18.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 10 client.4656.objecter ms_dispatch 0x7f9c94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:18.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:18.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:18.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T18:24:18.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c8404f030 con 0x7f9c940a9470 2024-03-25T18:24:18.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:18.718 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.716+0000 7f9c7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:18.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.917+0000 7f9c9a15e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:18.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.917+0000 7f9c9a15e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:18.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.917+0000 7f9c9a15e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:24:18.919 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.917+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c940aba50 con 0x7f9c940a9470 2024-03-25T18:24:18.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.978+0000 7f9c7bfff640 1 -- 172.21.15.87:0/1837873068 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f9c940aba50 con 0x7f9c940a9470 2024-03-25T18:24:18.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.978+0000 7f9c7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:18.980 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.978+0000 7f9c7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:18.980 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:18.980 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 10 librados: watch_flush enter 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 10 librados: watch_flush exit 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 10 client.4656.objecter successfully canceled tick 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 >> v1:172.21.15.171:6800/3417775508 conn(0x7f9c8006dc00 legacy=0x7f9c800700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 10 monclient: shutdown 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 >> v1:172.21.15.87:6789/0 conn(0x7f9c940a9470 legacy=0x7f9c940ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 shutdown_connections 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 >> 172.21.15.87:0/1837873068 conn(0x7f9c940617a0 msgr2=0x7f9c940b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 shutdown_connections 2024-03-25T18:24:18.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 -- 172.21.15.87:0/1837873068 wait complete. 2024-03-25T18:24:18.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:18.981+0000 7f9c9a15e640 1 librados: shutdown 2024-03-25T18:24:18.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:25.800 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.917+0000 7fa5bd396640 10 monclient: get_monmap_and_config 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.917+0000 7fa5bd396640 10 monclient: build_initial_monmap 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.917+0000 7fa5bd396640 10 monclient: monmap: 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:25.918259+0000 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:25.918259+0000 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:25.918 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:25.919 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:25.919 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:25.919 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:25.919 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 1 Processor -- start 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 1 -- start start 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient: init 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b80b4d60 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b80b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b814be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80af940 con 0x7fa5b814be90 2024-03-25T18:24:25.920 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80b0a70 con 0x7fa5b80b1730 2024-03-25T18:24:25.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80b0d50 con 0x7fa5b80b4d60 2024-03-25T18:24:25.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:25.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.918+0000 7fa5bd396640 10 monclient(hunting): authenticate will time out at 8381.213584s 2024-03-25T18:24:25.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.928+0000 7fa5b67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa5b80b1730 0x7fa5b80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:52952/0 (socket says 172.21.15.87:52952) 2024-03-25T18:24:25.929 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.928+0000 7fa5b67fc640 1 -- 172.21.15.87:0/2392466280 learned_addr learned my addr 172.21.15.87:0/2392466280 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771802905 0 0) 0x7fa5b80b0d50 con 0x7fa5b80b4d60 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a0003520 con 0x7fa5b80b4d60 2024-03-25T18:24:25.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936283465 0 0) 0x7fa5b80b0a70 con 0x7fa5b80b1730 2024-03-25T18:24:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:24:25.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.939+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80b0d50 con 0x7fa5b80b1730 2024-03-25T18:24:25.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728444284 0 0) 0x7fa5b80af940 con 0x7fa5b814be90 2024-03-25T18:24:25.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:25.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:25.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:25.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:24:25.948 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.946+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80b0a70 con 0x7fa5b814be90 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772930759 0 0) 0x7fa5b80b0d50 con 0x7fa5b80b1730 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 >> v1:172.21.15.87:6790/0 conn(0x7fa5b80b4d60 legacy=0x7fa5b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 >> v1:172.21.15.87:6789/0 conn(0x7fa5b814be90 legacy=0x7fa5b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80b1030 con 0x7fa5b80b1730 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5ac001670 con 0x7fa5b80b1730 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5bd396640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:25.959 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5b5ffb640 10 monclient: dump: 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:25.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.957+0000 7fa5bd396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ac0017d0 con 0x7fa5b80b1730 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 1 -- 172.21.15.87:0/2392466280 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5ac001be0 con 0x7fa5b80b1730 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5b5ffb640 10 monclient: dump: 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:25.968 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5bd396640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:25.969 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.967+0000 7fa5bd396640 4 set_mon_vals no callback set 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 10 monclient: shutdown 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 1 -- 172.21.15.87:0/2392466280 >> v1:172.21.15.171:6789/0 conn(0x7fa5b80b1730 legacy=0x7fa5b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 1 -- 172.21.15.87:0/2392466280 shutdown_connections 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 1 -- 172.21.15.87:0/2392466280 >> 172.21.15.87:0/2392466280 conn(0x7fa5b80617a0 msgr2=0x7fa5b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 1 -- 172.21.15.87:0/2392466280 shutdown_connections 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.968+0000 7fa5bd396640 1 -- 172.21.15.87:0/2392466280 wait complete. 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 10 monclient: build_initial_monmap 2024-03-25T18:24:25.970 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 10 monclient: monmap: 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 librados: starting msgr at 2024-03-25T18:24:25.971 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 librados: starting objecter 2024-03-25T18:24:25.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 Processor -- start 2024-03-25T18:24:25.972 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 -- start start 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 librados: setting wanted keys 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 1 librados: calling monclient init 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.969+0000 7fa5bd396640 10 monclient: init 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b80b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b80b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b814be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b81d7850 con 0x7fa5b80b4d60 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b81382f0 con 0x7fa5b80b1730 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b81d83f0 con 0x7fa5b814be90 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:25.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.970+0000 7fa5bd396640 10 monclient(hunting): authenticate will time out at 8381.265641s 2024-03-25T18:24:25.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.978+0000 7fa5b77fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fa5b814be90 0x7fa5b812fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:47064/0 (socket says 172.21.15.87:47064) 2024-03-25T18:24:25.978 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.978+0000 7fa5b77fe640 1 -- 172.21.15.87:0/2256823487 learned_addr learned my addr 172.21.15.87:0/2256823487 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.982+0000 7fa59f7fe640 10 client.?.objecter ms_handle_connect 0x7fa5b80b1730 2024-03-25T18:24:25.983 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.982+0000 7fa59f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:25.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.983+0000 7fa59f7fe640 10 client.?.objecter ms_handle_connect 0x7fa5b814be90 2024-03-25T18:24:25.984 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.983+0000 7fa59f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:25.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.991+0000 7fa59f7fe640 10 client.?.objecter ms_handle_connect 0x7fa5b80b4d60 2024-03-25T18:24:25.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.991+0000 7fa59f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:25.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657299846 0 0) 0x7fa5b81382f0 con 0x7fa5b80b1730 2024-03-25T18:24:25.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:25.997 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:25.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:25.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:24:25.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.996+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4004eb0 con 0x7fa5b80b1730 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033096720 0 0) 0x7fa5b81d83f0 con 0x7fa5b814be90 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:24:25.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:25.998+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b81382f0 con 0x7fa5b814be90 2024-03-25T18:24:26.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685568490 0 0) 0x7fa5b81d7850 con 0x7fa5b80b4d60 2024-03-25T18:24:26.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:26.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:26.008 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:26.008 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:24:26.008 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.006+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b81d83f0 con 0x7fa5b80b4d60 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1488054653 0 0) 0x7fa5a4004eb0 con 0x7fa5b80b1730 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b81d7850 con 0x7fa5b80b1730 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5a8001950 con 0x7fa5b80b1730 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.011 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:26.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.010+0000 7fa59f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:26.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660770425 0 0) 0x7fa5b81382f0 con 0x7fa5b814be90 2024-03-25T18:24:26.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5a4004eb0 con 0x7fa5b814be90 2024-03-25T18:24:26.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5b0001c50 con 0x7fa5b814be90 2024-03-25T18:24:26.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.012+0000 7fa59f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:26.014 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393784502 0 0) 0x7fa5b81d83f0 con 0x7fa5b80b4d60 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b81382f0 con 0x7fa5b80b4d60 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5ac001ae0 con 0x7fa5b80b4d60 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.026+0000 7fa59f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.029 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3530178935 0 0) 0x7fa5a4004eb0 con 0x7fa5b814be90 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 >> v1:172.21.15.171:6789/0 conn(0x7fa5b80b1730 legacy=0x7fa5b81d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 >> v1:172.21.15.87:6789/0 conn(0x7fa5b80b4d60 legacy=0x7fa5b8136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 1 monclient: found mon.c 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b8135c90 con 0x7fa5b814be90 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa59f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:26.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 5 monclient: authenticate success, global_id 4667 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 10 monclient: _renew_subs 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b8136e80 con 0x7fa5b814be90 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 10 monclient: _renew_subs 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b8143ae0 con 0x7fa5b814be90 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.027+0000 7fa5bd396640 1 librados: init done 2024-03-25T18:24:26.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.028+0000 7fa5bd396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.028+0000 7fa5bd396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.028+0000 7fa5bd396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:26.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.028+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b81370c0 con 0x7fa5b814be90 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5b00020e0 con 0x7fa5b814be90 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa5b0003bb0 con 0x7fa5b814be90 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:26.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 10 monclient: dump: 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:26.355 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa5b57fa640 4 set_mon_vals no callback set 2024-03-25T18:24:26.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.353+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa5b0003dd0 con 0x7fa5b814be90 2024-03-25T18:24:26.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.354+0000 7fa5b6ffd640 10 monclient: get_auth_request con 0x7fa5a406e110 auth_method 0 2024-03-25T18:24:26.356 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.354+0000 7fa59f7fe640 10 client.4667.objecter ms_handle_connect 0x7fa5a406e110 2024-03-25T18:24:26.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fa5b0054990 con 0x7fa5b814be90 2024-03-25T18:24:26.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 10 client.4667.objecter ms_dispatch 0x7fa5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 3 client.4667.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 3 client.4667.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5b00504a0 con 0x7fa5b814be90 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:26.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.360+0000 7fa59f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:26.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.584+0000 7fa5bd396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:26.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.584+0000 7fa5bd396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:26.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.584+0000 7fa5bd396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:26.585 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.584+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b81d7a00 con 0x7fa5b814be90 2024-03-25T18:24:26.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.681+0000 7fa59f7fe640 1 -- 172.21.15.87:0/2256823487 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fa5b81d7a00 con 0x7fa5b814be90 2024-03-25T18:24:26.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.681+0000 7fa59f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:26.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.681+0000 7fa59f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:26.683 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:26.683 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 10 librados: watch_flush enter 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 10 librados: watch_flush exit 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 10 client.4667.objecter successfully canceled tick 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa5a406e110 legacy=0x7fa5a40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 10 monclient: shutdown 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 >> v1:172.21.15.87:6790/0 conn(0x7fa5b814be90 legacy=0x7fa5b812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 shutdown_connections 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 >> 172.21.15.87:0/2256823487 conn(0x7fa5b80617a0 msgr2=0x7fa5b813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 shutdown_connections 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.683+0000 7fa5bd396640 1 -- 172.21.15.87:0/2256823487 wait complete. 2024-03-25T18:24:26.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:26.684+0000 7fa5bd396640 1 librados: shutdown 2024-03-25T18:24:26.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:33.231 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:20.944 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.279 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:20.993 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.313 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:21.026 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.347 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:21.010 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.347 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:21.014 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.347 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:21.045 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.347 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:21.059 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.364 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:21.043 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.389 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:21.102 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.503 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:33.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.609+0000 7f4c6bd72640 10 monclient: get_monmap_and_config 2024-03-25T18:24:33.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.609+0000 7f4c6bd72640 10 monclient: build_initial_monmap 2024-03-25T18:24:33.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.609+0000 7f4c6bd72640 10 monclient: monmap: 2024-03-25T18:24:33.610 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:33.610 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:33.610529+0000 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:33.610529+0000 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:33.611 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.610+0000 7f4c6bd72640 1 Processor -- start 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.610+0000 7f4c6bd72640 1 -- start start 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.610+0000 7f4c6bd72640 10 monclient: init 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-a con 0x7f4c640b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-c con 0x7f4c640b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6414be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:33.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640af940 con 0x7f4c640b4d60 2024-03-25T18:24:33.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640b0a70 con 0x7f4c640b1730 2024-03-25T18:24:33.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640b0d50 con 0x7f4c6414be90 2024-03-25T18:24:33.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:33.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.611+0000 7f4c6bd72640 10 monclient(hunting): authenticate will time out at 8388.906115s 2024-03-25T18:24:33.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.629+0000 7f4c6a30f640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4c6414be90 0x7f4c6414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:59548/0 (socket says 172.21.15.87:59548) 2024-03-25T18:24:33.630 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.629+0000 7f4c6a30f640 1 -- 172.21.15.87:0/28309493 learned_addr learned my addr 172.21.15.87:0/28309493 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407804097 0 0) 0x7f4c640af940 con 0x7f4c640b4d60 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:24:33.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.641+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50003520 con 0x7f4c640b4d60 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170439901 0 0) 0x7f4c640b0a70 con 0x7f4c640b1730 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:24:33.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.643+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c640af940 con 0x7f4c640b1730 2024-03-25T18:24:33.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587881566 0 0) 0x7f4c50003520 con 0x7f4c640b4d60 2024-03-25T18:24:33.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 >> v1:172.21.15.87:6790/0 conn(0x7f4c6414be90 legacy=0x7f4c6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.656 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 >> v1:172.21.15.171:6789/0 conn(0x7f4c640b1730 legacy=0x7f4c640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: _finish_hunting 0 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 monclient: found mon.noname-a 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640b1030 con 0x7f4c640b4d60 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: _finish_auth 0 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: _check_auth_tickets 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4c60001690 con 0x7f4c640b4d60 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c68b0c640 10 monclient: dump: 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.657 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c6bd72640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:24:33.658 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.655+0000 7f4c6bd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:33.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c600017f0 con 0x7f4c640b4d60 2024-03-25T18:24:33.670 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c68b0c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c68b0c640 1 -- 172.21.15.87:0/28309493 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4c60001c00 con 0x7f4c640b4d60 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c6bd72640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c6bd72640 4 set_mon_vals no callback set 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.669+0000 7f4c68b0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c68b0c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c68b0c640 10 monclient: dump: 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:33.671 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 10 monclient: shutdown 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 1 -- 172.21.15.87:0/28309493 >> v1:172.21.15.87:6789/0 conn(0x7f4c640b4d60 legacy=0x7f4c6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 1 -- 172.21.15.87:0/28309493 shutdown_connections 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 1 -- 172.21.15.87:0/28309493 >> 172.21.15.87:0/28309493 conn(0x7f4c640617a0 msgr2=0x7f4c64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 1 -- 172.21.15.87:0/28309493 shutdown_connections 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.670+0000 7f4c6bd72640 1 -- 172.21.15.87:0/28309493 wait complete. 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.671+0000 7f4c6bd72640 10 monclient: build_initial_monmap 2024-03-25T18:24:33.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.671+0000 7f4c6bd72640 10 monclient: monmap: 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.671+0000 7f4c6bd72640 1 librados: starting msgr at 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.671+0000 7f4c6bd72640 1 librados: starting objecter 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 Processor -- start 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 -- start start 2024-03-25T18:24:33.674 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 librados: setting wanted keys 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 librados: calling monclient init 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient: init 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-b con 0x7f4c640b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-a con 0x7f4c640b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6414be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c64136130 con 0x7f4c640b4d60 2024-03-25T18:24:33.675 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c64145800 con 0x7f4c640b1730 2024-03-25T18:24:33.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c641452b0 con 0x7f4c6414be90 2024-03-25T18:24:33.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:33.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.672+0000 7f4c6bd72640 10 monclient(hunting): authenticate will time out at 8388.967664s 2024-03-25T18:24:33.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.675+0000 7f4c69b0e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4c640b1730 0x7f4c641dac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:47178/0 (socket says 172.21.15.87:47178) 2024-03-25T18:24:33.676 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.675+0000 7f4c69b0e640 1 -- 172.21.15.87:0/1624729971 learned_addr learned my addr 172.21.15.87:0/1624729971 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:33.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.678+0000 7f4c5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c640b1730 2024-03-25T18:24:33.679 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.678+0000 7f4c5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:33.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100138591 0 0) 0x7f4c64145800 con 0x7f4c640b1730 2024-03-25T18:24:33.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:24:33.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.681+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c48004eb0 con 0x7f4c640b1730 2024-03-25T18:24:33.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.687+0000 7f4c5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c6414be90 2024-03-25T18:24:33.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.687+0000 7f4c5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:33.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.688+0000 7f4c5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c640b4d60 2024-03-25T18:24:33.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.688+0000 7f4c5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655374326 0 0) 0x7f4c64136130 con 0x7f4c640b4d60 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:24:33.699 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.698+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c64145800 con 0x7f4c640b4d60 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.700+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644874292 0 0) 0x7f4c48004eb0 con 0x7f4c640b1730 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c64136130 con 0x7f4c640b1730 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4c60001690 con 0x7f4c640b1730 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.702 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.703 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:33.703 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:33.703 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:33.703 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.701+0000 7f4c5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:33.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838811041 0 0) 0x7f4c641452b0 con 0x7f4c6414be90 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.706+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c48004eb0 con 0x7f4c6414be90 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2845633141 0 0) 0x7f4c64145800 con 0x7f4c640b4d60 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c641452b0 con 0x7f4c640b4d60 2024-03-25T18:24:33.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4c54001970 con 0x7f4c640b4d60 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.707+0000 7f4c5a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:33.709 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 423960115 0 0) 0x7f4c64136130 con 0x7f4c640b1730 2024-03-25T18:24:33.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 >> v1:172.21.15.87:6790/0 conn(0x7f4c6414be90 legacy=0x7f4c641e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 >> v1:172.21.15.87:6789/0 conn(0x7f4c640b4d60 legacy=0x7f4c641db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:33.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 1 monclient: found mon.b 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64145440 con 0x7f4c640b1730 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 10 monclient: _renew_subs 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c64135c10 con 0x7f4c640b1730 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 10 monclient: _renew_subs 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6413e3f0 con 0x7f4c640b1730 2024-03-25T18:24:33.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.713+0000 7f4c6bd72640 1 librados: init done 2024-03-25T18:24:33.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.714+0000 7f4c6bd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.714+0000 7f4c6bd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.714+0000 7f4c6bd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:33.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.714+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c641362c0 con 0x7f4c640b1730 2024-03-25T18:24:33.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.935+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c600038a0 con 0x7f4c640b1730 2024-03-25T18:24:33.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.935+0000 7f4c5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:33.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.935+0000 7f4c5bfff640 4 set_mon_vals no callback set 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.990+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4c60003c70 con 0x7f4c640b1730 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.990+0000 7f4c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.990+0000 7f4c5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.990+0000 7f4c5a7fc640 10 monclient: dump: 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:33.991 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:33.992 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.991+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4c60003e50 con 0x7f4c640b1730 2024-03-25T18:24:33.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.992+0000 7f4c6930d640 10 monclient: get_auth_request con 0x7f4c4806df20 auth_method 0 2024-03-25T18:24:33.993 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.992+0000 7f4c5a7fc640 10 client.4669.objecter ms_handle_connect 0x7f4c4806df20 2024-03-25T18:24:33.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.993+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7f4c600549b0 con 0x7f4c640b1730 2024-03-25T18:24:33.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.993+0000 7f4c5a7fc640 10 client.4669.objecter ms_dispatch 0x7f4c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:33.994 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.993+0000 7f4c5a7fc640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:33.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.993+0000 7f4c5a7fc640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:33.995 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.993+0000 7f4c5a7fc640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:24:33.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.998+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c600504c0 con 0x7f4c640b1730 2024-03-25T18:24:33.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.998+0000 7f4c5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:33.999 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:33.998+0000 7f4c5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:34.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.243+0000 7f4c6bd72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.244+0000 7f4c6bd72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.244+0000 7f4c6bd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:34.245 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.244+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c640627b0 con 0x7f4c640b1730 2024-03-25T18:24:34.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.371+0000 7f4c5a7fc640 1 -- 172.21.15.87:0/1624729971 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7f4c600523f0 con 0x7f4c640b1730 2024-03-25T18:24:34.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.371+0000 7f4c5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:34.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.371+0000 7f4c5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:34.373 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:34.373 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 10 librados: watch_flush enter 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 10 librados: watch_flush exit 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4c4806df20 legacy=0x7f4c480703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 10 monclient: shutdown 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 >> v1:172.21.15.171:6789/0 conn(0x7f4c640b1730 legacy=0x7f4c641dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 shutdown_connections 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 >> 172.21.15.87:0/1624729971 conn(0x7f4c640617a0 msgr2=0x7f4c640b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 shutdown_connections 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 -- 172.21.15.87:0/1624729971 wait complete. 2024-03-25T18:24:34.376 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:34.374+0000 7f4c6bd72640 1 librados: shutdown 2024-03-25T18:24:34.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:41.195 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.311+0000 7fc658954640 10 monclient: get_monmap_and_config 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.311+0000 7fc658954640 10 monclient: build_initial_monmap 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.311+0000 7fc658954640 10 monclient: monmap: 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.313 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:41.313273+0000 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:41.313273+0000 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:41.314 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 1 Processor -- start 2024-03-25T18:24:41.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 1 -- start start 2024-03-25T18:24:41.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient: init 2024-03-25T18:24:41.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient(hunting): picked mon.noname-a con 0x7fc6540b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient(hunting): picked mon.noname-c con 0x7fc6540b1730 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient(hunting): picked mon.noname-b con 0x7fc65414be90 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540af940 con 0x7fc6540b4d60 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540b0a70 con 0x7fc6540b1730 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540b0d50 con 0x7fc65414be90 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:41.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.312+0000 7fc658954640 10 monclient(hunting): authenticate will time out at 8396.608606s 2024-03-25T18:24:41.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.333+0000 7fc649d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc6540b1730 0x7fc6540b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:56722/0 (socket says 172.21.15.87:56722) 2024-03-25T18:24:41.335 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.333+0000 7fc649d9b640 1 -- 172.21.15.87:0/1800691120 learned_addr learned my addr 172.21.15.87:0/1800691120 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080727336 0 0) 0x7fc6540b0a70 con 0x7fc6540b1730 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:24:41.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.344+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc640003520 con 0x7fc6540b1730 2024-03-25T18:24:41.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462371820 0 0) 0x7fc6540b0d50 con 0x7fc65414be90 2024-03-25T18:24:41.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:24:41.351 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.348+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6540b0a70 con 0x7fc65414be90 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727629603 0 0) 0x7fc6540af940 con 0x7fc6540b4d60 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:24:41.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.356+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6540b0d50 con 0x7fc6540b4d60 2024-03-25T18:24:41.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023054508 0 0) 0x7fc640003520 con 0x7fc6540b1730 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 >> v1:172.21.15.87:6790/0 conn(0x7fc65414be90 legacy=0x7fc65414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 >> v1:172.21.15.87:6789/0 conn(0x7fc6540b4d60 legacy=0x7fc65413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540b1030 con 0x7fc6540b1730 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc63c001690 con 0x7fc6540b1730 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc651d9b640 10 monclient: dump: 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc658954640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:24:41.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.359+0000 7fc658954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:41.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc63c0017f0 con 0x7fc6540b1730 2024-03-25T18:24:41.384 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 1 -- 172.21.15.87:0/1800691120 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc63c001c00 con 0x7fc6540b1730 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc658954640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc658954640 4 set_mon_vals no callback set 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.382+0000 7fc651d9b640 10 monclient: dump: 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:41.385 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:41.386 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 10 monclient: shutdown 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 1 -- 172.21.15.87:0/1800691120 >> v1:172.21.15.171:6789/0 conn(0x7fc6540b1730 legacy=0x7fc6540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 1 -- 172.21.15.87:0/1800691120 shutdown_connections 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 1 -- 172.21.15.87:0/1800691120 >> 172.21.15.87:0/1800691120 conn(0x7fc6540617a0 msgr2=0x7fc654061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 1 -- 172.21.15.87:0/1800691120 shutdown_connections 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.383+0000 7fc658954640 1 -- 172.21.15.87:0/1800691120 wait complete. 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.384+0000 7fc658954640 10 monclient: build_initial_monmap 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.384+0000 7fc658954640 10 monclient: monmap: 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:41.387 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 librados: starting msgr at 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 librados: starting objecter 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 Processor -- start 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 -- start start 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 librados: setting wanted keys 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 1 librados: calling monclient init 2024-03-25T18:24:41.388 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.385+0000 7fc658954640 10 monclient: init 2024-03-25T18:24:41.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:41.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:41.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient(hunting): picked mon.noname-c con 0x7fc6540b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:41.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient(hunting): picked mon.noname-b con 0x7fc6540b4d60 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:41.389 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient(hunting): picked mon.noname-a con 0x7fc65414be90 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:41.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc654141790 con 0x7fc65414be90 2024-03-25T18:24:41.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc654141a00 con 0x7fc6540b4d60 2024-03-25T18:24:41.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6541d7a90 con 0x7fc6540b1730 2024-03-25T18:24:41.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:41.390 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.386+0000 7fc658954640 10 monclient(hunting): authenticate will time out at 8396.682148s 2024-03-25T18:24:41.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.391+0000 7fc649d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc6540b4d60 0x7fc654142a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:56766/0 (socket says 172.21.15.87:56766) 2024-03-25T18:24:41.393 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.391+0000 7fc649d9b640 1 -- 172.21.15.87:0/3510103414 learned_addr learned my addr 172.21.15.87:0/3510103414 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:41.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.396+0000 7fc64b7fe640 10 client.?.objecter ms_handle_connect 0x7fc6540b4d60 2024-03-25T18:24:41.398 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.396+0000 7fc64b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.401+0000 7fc64b7fe640 10 client.?.objecter ms_handle_connect 0x7fc65414be90 2024-03-25T18:24:41.403 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.401+0000 7fc64b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282348249 0 0) 0x7fc654141a00 con 0x7fc6540b4d60 2024-03-25T18:24:41.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.404 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:24:41.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.402+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc634004eb0 con 0x7fc6540b4d60 2024-03-25T18:24:41.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.403+0000 7fc64b7fe640 10 client.?.objecter ms_handle_connect 0x7fc6540b1730 2024-03-25T18:24:41.405 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.403+0000 7fc64b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350251564 0 0) 0x7fc654141790 con 0x7fc65414be90 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:24:41.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.406+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654141a00 con 0x7fc65414be90 2024-03-25T18:24:41.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482277953 0 0) 0x7fc6541d7a90 con 0x7fc6540b1730 2024-03-25T18:24:41.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:41.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:41.412 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:41.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:24:41.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654141790 con 0x7fc6540b1730 2024-03-25T18:24:41.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000251068 0 0) 0x7fc634004eb0 con 0x7fc6540b4d60 2024-03-25T18:24:41.413 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.410+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6541d7a90 con 0x7fc6540b4d60 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.412+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc63c001c60 con 0x7fc6540b4d60 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.412+0000 7fc64b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.412+0000 7fc64b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.412+0000 7fc64b7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:41.414 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.415 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.412+0000 7fc64b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:41.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1646172798 0 0) 0x7fc654141a00 con 0x7fc65414be90 2024-03-25T18:24:41.417 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc634004eb0 con 0x7fc65414be90 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc64c001b50 con 0x7fc65414be90 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.415+0000 7fc64b7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:41.418 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2411129772 0 0) 0x7fc6541d7a90 con 0x7fc6540b4d60 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 >> v1:172.21.15.87:6790/0 conn(0x7fc6540b1730 legacy=0x7fc65413b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 >> v1:172.21.15.87:6789/0 conn(0x7fc65414be90 legacy=0x7fc6541d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 1 monclient: found mon.b 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc654143140 con 0x7fc6540b4d60 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:41.424 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc64b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 10 monclient: _renew_subs 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6541d86c0 con 0x7fc6540b4d60 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 10 monclient: _renew_subs 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6541422e0 con 0x7fc6540b4d60 2024-03-25T18:24:41.425 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.421+0000 7fc658954640 1 librados: init done 2024-03-25T18:24:41.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.423+0000 7fc658954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.423+0000 7fc658954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.423+0000 7fc658954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.426 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.423+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6540627b0 con 0x7fc6540b4d60 2024-03-25T18:24:41.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc63c0011f0 con 0x7fc6540b4d60 2024-03-25T18:24:41.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:41.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fc63c004cb0 con 0x7fc6540b4d60 2024-03-25T18:24:41.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc64b7fe640 10 monclient: dump: 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:41.709 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.706+0000 7fc65159a640 4 set_mon_vals no callback set 2024-03-25T18:24:41.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.707+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fc63c01ad00 con 0x7fc6540b4d60 2024-03-25T18:24:41.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.708+0000 7fc65259c640 10 monclient: get_auth_request con 0x7fc63406dec0 auth_method 0 2024-03-25T18:24:41.710 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.708+0000 7fc64b7fe640 10 client.4675.objecter ms_handle_connect 0x7fc63406dec0 2024-03-25T18:24:41.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.712+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 953519888 0 0) 0x7fc63c054a20 con 0x7fc6540b4d60 2024-03-25T18:24:41.714 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.712+0000 7fc64b7fe640 10 client.4675.objecter ms_dispatch 0x7fc654000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.712+0000 7fc64b7fe640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.712+0000 7fc64b7fe640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.712+0000 7fc64b7fe640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.713+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc63c050360 con 0x7fc6540b4d60 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.713+0000 7fc64b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:41.715 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.713+0000 7fc64b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:41.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.941+0000 7fc658954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:41.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.941+0000 7fc658954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:41.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.941+0000 7fc658954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:41.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:41.941+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc65414bbc0 con 0x7fc6540b4d60 2024-03-25T18:24:42.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.024+0000 7fc64b7fe640 1 -- 172.21.15.87:0/3510103414 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2768680031) 0x7fc63c052460 con 0x7fc6540b4d60 2024-03-25T18:24:42.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.024+0000 7fc64b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:42.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.024+0000 7fc64b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:42.026 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:42.026 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:14:44.644650+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:44.644650+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.026+0000 7fc658954640 10 librados: watch_flush enter 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 10 librados: watch_flush exit 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 >> v1:172.21.15.171:6800/3417775508 conn(0x7fc63406dec0 legacy=0x7fc634070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 10 monclient: shutdown 2024-03-25T18:24:42.029 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 >> v1:172.21.15.171:6789/0 conn(0x7fc6540b4d60 legacy=0x7fc654142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 shutdown_connections 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 >> 172.21.15.87:0/3510103414 conn(0x7fc6540617a0 msgr2=0x7fc6540b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 shutdown_connections 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 -- 172.21.15.87:0/3510103414 wait complete. 2024-03-25T18:24:42.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:42.027+0000 7fc658954640 1 librados: shutdown 2024-03-25T18:24:42.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:48.846 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.960+0000 7fa930a7b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.960+0000 7fa930a7b640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.960+0000 7fa930a7b640 10 monclient: monmap: 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:48.961061+0000 2024-03-25T18:24:48.962 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:48.961061+0000 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.960+0000 7fa930a7b640 1 Processor -- start 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 1 -- start start 2024-03-25T18:24:48.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient: init 2024-03-25T18:24:48.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:48.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa92c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:48.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa92c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa92c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.961+0000 7fa930a7b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0af940 con 0x7fa92c0b4d60 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.962+0000 7fa930a7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0b0a70 con 0x7fa92c14be90 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.962+0000 7fa930a7b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0b0d50 con 0x7fa92c0b1730 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.962+0000 7fa930a7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.965 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.962+0000 7fa930a7b640 10 monclient(hunting): authenticate will time out at 8404.256907s 2024-03-25T18:24:48.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.978+0000 7fa929d9b640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fa92c0b1730 0x7fa92c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:53994/0 (socket says 172.21.15.87:53994) 2024-03-25T18:24:48.979 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.978+0000 7fa929d9b640 1 -- 172.21.15.87:0/931046180 learned_addr learned my addr 172.21.15.87:0/931046180 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145341031 0 0) 0x7fa92c0b0a70 con 0x7fa92c14be90 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:24:48.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:48.997+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914003520 con 0x7fa92c14be90 2024-03-25T18:24:49.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636192221 0 0) 0x7fa92c0af940 con 0x7fa92c0b4d60 2024-03-25T18:24:49.001 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:49.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:49.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:49.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:24:49.002 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.000+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c0b0a70 con 0x7fa92c0b4d60 2024-03-25T18:24:49.005 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.004+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155203555 0 0) 0x7fa92c0b0d50 con 0x7fa92c0b1730 2024-03-25T18:24:49.005 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.004+0000 7fa92959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:49.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.004+0000 7fa92959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:49.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.004+0000 7fa92959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:49.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.004+0000 7fa92959a640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:24:49.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.005+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c0af940 con 0x7fa92c0b1730 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867659557 0 0) 0x7fa914003520 con 0x7fa92c14be90 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 >> v1:172.21.15.87:6790/0 conn(0x7fa92c0b1730 legacy=0x7fa92c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 >> v1:172.21.15.87:6789/0 conn(0x7fa92c0b4d60 legacy=0x7fa92c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 monclient: found mon.noname-c 2024-03-25T18:24:49.011 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0b1030 con 0x7fa92c14be90 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: _finish_auth 0 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa924001a10 con 0x7fa92c14be90 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa92959a640 10 monclient: dump: 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:49.012 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:49.013 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:49.013 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:49.013 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa930a7b640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:24:49.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.010+0000 7fa930a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:49.026 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa924001b70 con 0x7fa92c14be90 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 1 -- 172.21.15.87:0/931046180 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa924001f80 con 0x7fa92c14be90 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa92959a640 10 monclient: dump: 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:49.027 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:49.028 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:49.028 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa930a7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:49.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.025+0000 7fa930a7b640 4 set_mon_vals no callback set 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.026+0000 7fa930a7b640 10 monclient: shutdown 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.026+0000 7fa930a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.026+0000 7fa930a7b640 1 -- 172.21.15.87:0/931046180 >> v1:172.21.15.171:6789/0 conn(0x7fa92c14be90 legacy=0x7fa92c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 1 -- 172.21.15.87:0/931046180 shutdown_connections 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 1 -- 172.21.15.87:0/931046180 >> 172.21.15.87:0/931046180 conn(0x7fa92c0617a0 msgr2=0x7fa92c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 1 -- 172.21.15.87:0/931046180 shutdown_connections 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 1 -- 172.21.15.87:0/931046180 wait complete. 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 10 monclient: build_initial_monmap 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.027+0000 7fa930a7b640 10 monclient: monmap: 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:49.030 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 librados: starting msgr at 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 librados: starting objecter 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 Processor -- start 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 -- start start 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 librados: setting wanted keys 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 1 librados: calling monclient init 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.028+0000 7fa930a7b640 10 monclient: init 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:49.031 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa92c0b1730 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa92c0b4d60 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa92c14be90 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c135ee0 con 0x7fa92c0b4d60 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c135b00 con 0x7fa92c14be90 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c135c90 con 0x7fa92c0b1730 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:49.032 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.029+0000 7fa930a7b640 10 monclient(hunting): authenticate will time out at 8404.323998s 2024-03-25T18:24:49.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.038+0000 7fa92a59c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fa92c0b1730 0x7fa92c12fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:54016/0 (socket says 172.21.15.87:54016) 2024-03-25T18:24:49.039 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.038+0000 7fa92a59c640 1 -- 172.21.15.87:0/3185308968 learned_addr learned my addr 172.21.15.87:0/3185308968 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:49.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.040+0000 7fa91affd640 10 client.?.objecter ms_handle_connect 0x7fa92c0b1730 2024-03-25T18:24:49.040 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.040+0000 7fa91affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:49.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.045+0000 7fa91affd640 10 client.?.objecter ms_handle_connect 0x7fa92c14be90 2024-03-25T18:24:49.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.045+0000 7fa91affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:49.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413629991 0 0) 0x7fa92c135c90 con 0x7fa92c0b1730 2024-03-25T18:24:49.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:49.050 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:49.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:49.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:24:49.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.049+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa92c0b1730 2024-03-25T18:24:49.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.053+0000 7fa91affd640 10 client.?.objecter ms_handle_connect 0x7fa92c0b4d60 2024-03-25T18:24:49.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.053+0000 7fa91affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991839896 0 0) 0x7fa92c135b00 con 0x7fa92c14be90 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:24:49.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.054+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c135c90 con 0x7fa92c14be90 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571559013 0 0) 0x7fa90c004eb0 con 0x7fa92c0b1730 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa92c135b00 con 0x7fa92c0b1730 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa91c001a30 con 0x7fa92c0b1730 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 10 monclient(hunting): dump: 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:49.062 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.063 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.061+0000 7fa91affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:49.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1721451486 0 0) 0x7fa92c135c90 con 0x7fa92c14be90 2024-03-25T18:24:49.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa92c14be90 2024-03-25T18:24:49.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa924001e80 con 0x7fa92c14be90 2024-03-25T18:24:49.066 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.065+0000 7fa91affd640 10 monclient(hunting): dump: 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:49.067 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620222447 0 0) 0x7fa92c135ee0 con 0x7fa92c0b4d60 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:24:49.074 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.073+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c135c90 con 0x7fa92c0b4d60 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3589849826 0 0) 0x7fa92c135b00 con 0x7fa92c0b1730 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 >> v1:172.21.15.171:6789/0 conn(0x7fa92c14be90 legacy=0x7fa92c1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 >> v1:172.21.15.87:6789/0 conn(0x7fa92c0b4d60 legacy=0x7fa92c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 1 monclient: found mon.c 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c136960 con 0x7fa92c0b1730 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 10 monclient: _finish_auth 0 2024-03-25T18:24:49.077 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa930a7b640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa930a7b640 10 monclient: _renew_subs 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa930a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.075+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c130470 con 0x7fa92c0b1730 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 10 monclient: _renew_subs 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c13e870 con 0x7fa92c0b1730 2024-03-25T18:24:49.078 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 1 librados: init done 2024-03-25T18:24:49.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:49.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.076+0000 7fa930a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:49.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.077+0000 7fa930a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:49.079 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.077+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92c130810 con 0x7fa92c0b1730 2024-03-25T18:24:49.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa91c001c90 con 0x7fa92c0b1730 2024-03-25T18:24:49.379 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa91c0038c0 con 0x7fa92c0b1730 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa91affd640 10 monclient: dump: 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:49.380 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:49.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.378+0000 7fa928d99640 4 set_mon_vals no callback set 2024-03-25T18:24:49.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.379+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa91c003ae0 con 0x7fa92c0b1730 2024-03-25T18:24:49.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.380+0000 7fa929d9b640 10 monclient: get_auth_request con 0x7fa90c06df50 auth_method 0 2024-03-25T18:24:49.381 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.380+0000 7fa91affd640 10 client.4685.objecter ms_handle_connect 0x7fa90c06df50 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7fa91c0546f0 con 0x7fa92c0b1730 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 10 client.4685.objecter ms_dispatch 0x7fa92c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa91c050200 con 0x7fa92c0b1730 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:49.386 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.385+0000 7fa91affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:49.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.631+0000 7fa930a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.631+0000 7fa930a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.631+0000 7fa930a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:24:49.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.631+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa92c136af0 con 0x7fa92c0b1730 2024-03-25T18:24:49.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.754+0000 7fa91affd640 1 -- 172.21.15.87:0/3185308968 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7fa91c052130 con 0x7fa92c0b1730 2024-03-25T18:24:49.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.754+0000 7fa91affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.754+0000 7fa91affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:49.756 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:49.756 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:49.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 10 librados: watch_flush enter 2024-03-25T18:24:49.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 10 librados: watch_flush exit 2024-03-25T18:24:49.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-25T18:24:49.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 10 client.4685.objecter successfully canceled tick 2024-03-25T18:24:49.758 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa90c06df50 legacy=0x7fa90c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 10 monclient: shutdown 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 >> v1:172.21.15.87:6790/0 conn(0x7fa92c0b1730 legacy=0x7fa92c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 shutdown_connections 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.756+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 >> 172.21.15.87:0/3185308968 conn(0x7fa92c0617a0 msgr2=0x7fa92c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.757+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 shutdown_connections 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.757+0000 7fa930a7b640 1 -- 172.21.15.87:0/3185308968 wait complete. 2024-03-25T18:24:49.759 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:49.757+0000 7fa930a7b640 1 librados: shutdown 2024-03-25T18:24:49.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:53.326 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:41.039 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.370 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:41.084 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.396 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:10:41.110 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.432 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:41.110 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.437 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:41.115 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.449 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:41.162 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.457 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:41.170 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.474 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:10:41.153 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.494 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:10:41.207 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.576 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 10 monclient: get_monmap_and_config 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 10 monclient: build_initial_monmap 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 10 monclient: monmap: 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:24:56.687880+0000 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:24:56.687880+0000 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:56.688 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:56.689 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:56.689 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:24:56.689 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:24:56.689 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 1 Processor -- start 2024-03-25T18:24:56.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 1 -- start start 2024-03-25T18:24:56.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.687+0000 7f1bf95ef640 10 monclient: init 2024-03-25T18:24:56.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:56.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4142520 con 0x7f1bf41430c0 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b28a0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b2a30 con 0x7f1bf413f710 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.688+0000 7f1bf95ef640 10 monclient(hunting): authenticate will time out at 8411.983289s 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.689+0000 7f1bf27fc640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f1bf413f710 0x7f1bf41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:58832/0 (socket says 172.21.15.87:58832) 2024-03-25T18:24:56.691 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.689+0000 7f1bf27fc640 1 -- 172.21.15.87:0/773424368 learned_addr learned my addr 172.21.15.87:0/773424368 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:56.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912770857 0 0) 0x7f1bf40b28a0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:56.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:56.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:56.760 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:24:56.761 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.759+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bdc003520 con 0x7f1bf40a98f0 2024-03-25T18:24:56.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 845723751 0 0) 0x7f1bdc003520 con 0x7f1bf40a98f0 2024-03-25T18:24:56.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 >> v1:172.21.15.87:6790/0 conn(0x7f1bf413f710 legacy=0x7f1bf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 >> v1:172.21.15.87:6789/0 conn(0x7f1bf41430c0 legacy=0x7f1bf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.771 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40b44c0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1be4001690 con 0x7f1bf40a98f0 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf1ffb640 10 monclient: dump: 2024-03-25T18:24:56.772 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf95ef640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:24:56.773 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.770+0000 7f1bf95ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:56.782 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be40017f0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 1 -- 172.21.15.87:0/773424368 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1be4001c40 con 0x7f1bf40a98f0 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf95ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf95ef640 4 set_mon_vals no callback set 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf1ffb640 10 monclient: dump: 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:56.783 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:56.784 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.786 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf95ef640 10 monclient: shutdown 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf95ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.782+0000 7f1bf95ef640 1 -- 172.21.15.87:0/773424368 >> v1:172.21.15.171:6789/0 conn(0x7f1bf40a98f0 legacy=0x7f1bf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 1 -- 172.21.15.87:0/773424368 shutdown_connections 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 1 -- 172.21.15.87:0/773424368 >> 172.21.15.87:0/773424368 conn(0x7f1bf40b7ea0 msgr2=0x7f1bf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 1 -- 172.21.15.87:0/773424368 shutdown_connections 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 1 -- 172.21.15.87:0/773424368 wait complete. 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 10 monclient: build_initial_monmap 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.783+0000 7f1bf95ef640 10 monclient: monmap: 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:24:56.787 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 librados: starting msgr at 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 librados: starting objecter 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 Processor -- start 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 -- start start 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 librados: setting wanted keys 2024-03-25T18:24:56.788 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 1 librados: calling monclient init 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 10 monclient: init 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf40a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.784+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf413be70 con 0x7f1bf41430c0 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4131660 con 0x7f1bf40a98f0 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf41317f0 con 0x7f1bf413f710 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:56.789 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.785+0000 7f1bf95ef640 10 monclient(hunting): authenticate will time out at 8412.079899s 2024-03-25T18:24:56.806 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.805+0000 7f1bf2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1bf40a98f0 0x7f1bf4130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:50220/0 (socket says 172.21.15.87:50220) 2024-03-25T18:24:56.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.805+0000 7f1bf2ffd640 1 -- 172.21.15.87:0/3096779039 learned_addr learned my addr 172.21.15.87:0/3096779039 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:24:56.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.806+0000 7f1bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1bf40a98f0 2024-03-25T18:24:56.807 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.806+0000 7f1bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:56.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.811+0000 7f1bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1bf41430c0 2024-03-25T18:24:56.811 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.811+0000 7f1bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:56.818 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.817+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635115938 0 0) 0x7f1bf4131660 con 0x7f1bf40a98f0 2024-03-25T18:24:56.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.818+0000 7f1bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:56.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.818+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:56.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.818+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:56.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.818+0000 7f1bdb7fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:24:56.819 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.818+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be0004eb0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.830 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.830+0000 7f1bdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1bf413f710 2024-03-25T18:24:56.831 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.830+0000 7f1bdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:56.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334122462 0 0) 0x7f1bf413be70 con 0x7f1bf41430c0 2024-03-25T18:24:56.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:56.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:56.841 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:56.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:24:56.842 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.840+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4131660 con 0x7f1bf41430c0 2024-03-25T18:24:56.844 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687452493 0 0) 0x7f1be0004eb0 con 0x7f1bf40a98f0 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf413be70 con 0x7f1bf40a98f0 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1be4001c00 con 0x7f1bf40a98f0 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:56.845 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:56.846 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:56.846 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:56.846 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.846 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.843+0000 7f1bdb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:56.847 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644805645 0 0) 0x7f1bf41317f0 con 0x7f1bf413f710 2024-03-25T18:24:56.847 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:56.847 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:56.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:56.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:24:56.848 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.846+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be0004eb0 con 0x7f1bf413f710 2024-03-25T18:24:56.860 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.859+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932601404 0 0) 0x7f1bf4131660 con 0x7f1bf41430c0 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.859+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf41317f0 con 0x7f1bf41430c0 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1bec001b10 con 0x7f1bf41430c0 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:56.861 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:56.862 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3471595511 0 0) 0x7f1bf413be70 con 0x7f1bf40a98f0 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 >> v1:172.21.15.87:6790/0 conn(0x7f1bf413f710 legacy=0x7f1bf41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 >> v1:172.21.15.87:6789/0 conn(0x7f1bf41430c0 legacy=0x7f1bf41e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 monclient: found mon.b 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4131980 con 0x7f1bf40a98f0 2024-03-25T18:24:56.863 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 10 monclient: _renew_subs 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf4135c60 con 0x7f1bf40a98f0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 10 monclient: _renew_subs 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf40b7770 con 0x7f1bf40a98f0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.860+0000 7f1bf95ef640 1 librados: init done 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.861+0000 7f1bf95ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.861+0000 7f1bf95ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.861+0000 7f1bf95ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:56.864 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:56.861+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf41371d0 con 0x7f1bf40a98f0 2024-03-25T18:24:57.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be4003960 con 0x7f1bf40a98f0 2024-03-25T18:24:57.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:57.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f1be4003cf0 con 0x7f1bf40a98f0 2024-03-25T18:24:57.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:57.147 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 10 monclient: dump: 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bf17fa640 4 set_mon_vals no callback set 2024-03-25T18:24:57.148 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.146+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f1be4003ed0 con 0x7f1bf40a98f0 2024-03-25T18:24:57.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.147+0000 7f1bf27fc640 10 monclient: get_auth_request con 0x7f1be006e040 auth_method 0 2024-03-25T18:24:57.149 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.147+0000 7f1bdb7fe640 10 client.4687.objecter ms_handle_connect 0x7f1be006e040 2024-03-25T18:24:57.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.153+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f1be4054a10 con 0x7f1bf40a98f0 2024-03-25T18:24:57.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.153+0000 7f1bdb7fe640 10 client.4687.objecter ms_dispatch 0x7f1bf4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:57.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.153+0000 7f1bdb7fe640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:57.154 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.153+0000 7f1bdb7fe640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:57.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.153+0000 7f1bdb7fe640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:24:57.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.155+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1be4050520 con 0x7f1bf40a98f0 2024-03-25T18:24:57.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.155+0000 7f1bdb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:57.155 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.155+0000 7f1bdb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:57.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.405+0000 7f1bf95ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:57.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.405+0000 7f1bf95ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:57.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.405+0000 7f1bf95ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:24:57.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.405+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf4074cc0 con 0x7f1bf40a98f0 2024-03-25T18:24:57.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.430+0000 7f1bdb7fe640 1 -- 172.21.15.87:0/3096779039 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f1be4052450 con 0x7f1bf40a98f0 2024-03-25T18:24:57.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.430+0000 7f1bdb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:57.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.430+0000 7f1bdb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:57.431 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:24:57.431 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:57.433 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 10 librados: watch_flush enter 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 10 librados: watch_flush exit 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 >> v1:172.21.15.171:6800/3417775508 conn(0x7f1be006e040 legacy=0x7f1be0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 10 monclient: shutdown 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 >> v1:172.21.15.171:6789/0 conn(0x7f1bf40a98f0 legacy=0x7f1bf4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 shutdown_connections 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 >> 172.21.15.87:0/3096779039 conn(0x7f1bf40b7ea0 msgr2=0x7f1bf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 shutdown_connections 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 -- 172.21.15.87:0/3096779039 wait complete. 2024-03-25T18:24:57.434 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:24:57.432+0000 7f1bf95ef640 1 librados: shutdown 2024-03-25T18:24:57.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:04.252 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:04.360 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.359+0000 7fa8b7fd0640 10 monclient: get_monmap_and_config 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.359+0000 7fa8b7fd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.359+0000 7fa8b7fd0640 10 monclient: monmap: 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:04.360738+0000 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:04.360738+0000 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:04.361 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.362 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.360+0000 7fa8b7fd0640 1 Processor -- start 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.360+0000 7fa8b7fd0640 1 -- start start 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.360+0000 7fa8b7fd0640 10 monclient: init 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b00a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b01430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b0142520 con 0x7fa8b01430c0 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b28a0 con 0x7fa8b013f710 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b2a30 con 0x7fa8b00a98f0 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:04.363 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.361+0000 7fa8b7fd0640 10 monclient(hunting): authenticate will time out at 8419.656310s 2024-03-25T18:25:04.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.370+0000 7fa8b656d640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa8b01430c0 0x7fa8b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:37230/0 (socket says 172.21.15.87:37230) 2024-03-25T18:25:04.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.370+0000 7fa8b656d640 1 -- 172.21.15.87:0/3095660747 learned_addr learned my addr 172.21.15.87:0/3095660747 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:04.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803724895 0 0) 0x7fa8b0142520 con 0x7fa8b01430c0 2024-03-25T18:25:04.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.382 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:25:04.383 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.381+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c003520 con 0x7fa8b01430c0 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799554485 0 0) 0x7fa8b00b28a0 con 0x7fa8b013f710 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:25:04.385 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.384+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0142520 con 0x7fa8b013f710 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251537479 0 0) 0x7fa8b0142520 con 0x7fa8b013f710 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 >> v1:172.21.15.87:6790/0 conn(0x7fa8b00a98f0 legacy=0x7fa8b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 >> v1:172.21.15.87:6789/0 conn(0x7fa8b01430c0 legacy=0x7fa8b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 monclient: found mon.noname-c 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:04.406 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00b44c0 con 0x7fa8b013f710 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: _finish_auth 0 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa8a0001670 con 0x7fa8b013f710 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b4d6a640 10 monclient: dump: 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.407 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b7fd0640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:25:04.408 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.405+0000 7fa8b7fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8a00017d0 con 0x7fa8b013f710 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 1 -- 172.21.15.87:0/3095660747 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa8a0001ba0 con 0x7fa8b013f710 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b7fd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b7fd0640 4 set_mon_vals no callback set 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.415+0000 7fa8b4d6a640 10 monclient: dump: 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:04.416 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:04.417 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 10 monclient: shutdown 2024-03-25T18:25:04.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:04.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3095660747 >> v1:172.21.15.171:6789/0 conn(0x7fa8b013f710 legacy=0x7fa8b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3095660747 shutdown_connections 2024-03-25T18:25:04.419 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3095660747 >> 172.21.15.87:0/3095660747 conn(0x7fa8b00b7ea0 msgr2=0x7fa8b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3095660747 shutdown_connections 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.416+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3095660747 wait complete. 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 10 monclient: monmap: 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.420 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 librados: starting msgr at 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 librados: starting objecter 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 Processor -- start 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 -- start start 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 librados: setting wanted keys 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 1 librados: calling monclient init 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.417+0000 7fa8b7fd0640 10 monclient: init 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b00a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b013f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b01430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b01db660 con 0x7fa8b013f710 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b01db7f0 con 0x7fa8b01430c0 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b014dee0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:04.421 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.418+0000 7fa8b7fd0640 10 monclient(hunting): authenticate will time out at 8419.713214s 2024-03-25T18:25:04.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.426+0000 7fa8b556b640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fa8b013f710 0x7fa8b00ae760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:37258/0 (socket says 172.21.15.87:37258) 2024-03-25T18:25:04.427 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.426+0000 7fa8b556b640 1 -- 172.21.15.87:0/3933747014 learned_addr learned my addr 172.21.15.87:0/3933747014 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:04.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.430+0000 7fa8a67fc640 10 client.?.objecter ms_handle_connect 0x7fa8b013f710 2024-03-25T18:25:04.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.430+0000 7fa8a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.431+0000 7fa8a67fc640 10 client.?.objecter ms_handle_connect 0x7fa8b00a98f0 2024-03-25T18:25:04.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.431+0000 7fa8a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.434+0000 7fa8a67fc640 10 client.?.objecter ms_handle_connect 0x7fa8b01430c0 2024-03-25T18:25:04.435 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.434+0000 7fa8a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:04.444 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.443+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393646916 0 0) 0x7fa8b014dee0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.444+0000 7fa8a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.444+0000 7fa8a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.444+0000 7fa8a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.444+0000 7fa8a67fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:25:04.445 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.444+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395565397 0 0) 0x7fa8b01db7f0 con 0x7fa8b01430c0 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:25:04.446 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.445+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b014dee0 con 0x7fa8b01430c0 2024-03-25T18:25:04.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 653293072 0 0) 0x7fa894004eb0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b01db7f0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa8ac001930 con 0x7fa8b00a98f0 2024-03-25T18:25:04.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.459 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 10 monclient(hunting): dump: 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.460 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.458+0000 7fa8a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022202960 0 0) 0x7fa8b01db660 con 0x7fa8b013f710 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:25:04.461 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.459+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b013f710 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.467+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4146257376 0 0) 0x7fa8b01db7f0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 >> v1:172.21.15.171:6789/0 conn(0x7fa8b01430c0 legacy=0x7fa8b01e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 >> v1:172.21.15.87:6789/0 conn(0x7fa8b013f710 legacy=0x7fa8b00ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 1 monclient: found mon.c 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:04.469 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b014e070 con 0x7fa8b00a98f0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 10 monclient: _renew_subs 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b00b11e0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 10 monclient: _renew_subs 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b01e3470 con 0x7fa8b00a98f0 2024-03-25T18:25:04.470 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.468+0000 7fa8b7fd0640 1 librados: init done 2024-03-25T18:25:04.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.469+0000 7fa8b7fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:04.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.469+0000 7fa8b7fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:04.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.469+0000 7fa8b7fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:04.471 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.469+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b014e260 con 0x7fa8b00a98f0 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.781+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8ac0035f0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.781+0000 7fa8a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.781+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fa8ac003940 con 0x7fa8b00a98f0 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.781+0000 7fa8a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.782+0000 7fa8a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.782+0000 7fa8a67fc640 10 monclient: dump: 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.783 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.782+0000 7fa8a7fff640 4 set_mon_vals no callback set 2024-03-25T18:25:04.784 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.782+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fa8ac003b60 con 0x7fa8b00a98f0 2024-03-25T18:25:04.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.783+0000 7fa8b556b640 10 monclient: get_auth_request con 0x7fa89406de00 auth_method 0 2024-03-25T18:25:04.785 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.784+0000 7fa8a67fc640 10 client.4697.objecter ms_handle_connect 0x7fa89406de00 2024-03-25T18:25:04.791 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7fa8ac0535f0 con 0x7fa8b00a98f0 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 10 client.4697.objecter ms_dispatch 0x7fa8b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8ac04f100 con 0x7fa8b00a98f0 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:04.792 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:04.790+0000 7fa8a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:05.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.051+0000 7fa8b7fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.052+0000 7fa8b7fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.052+0000 7fa8b7fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:05.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.052+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b00b1760 con 0x7fa8b00a98f0 2024-03-25T18:25:05.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.100+0000 7fa8a67fc640 1 -- 172.21.15.87:0/3933747014 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7fa8ac051030 con 0x7fa8b00a98f0 2024-03-25T18:25:05.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.100+0000 7fa8a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.102 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.100+0000 7fa8a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:05.102 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:05.103 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 10 librados: watch_flush enter 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 10 librados: watch_flush exit 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 10 client.4697.objecter successfully canceled tick 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 >> v1:172.21.15.171:6800/3417775508 conn(0x7fa89406de00 legacy=0x7fa8940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 10 monclient: shutdown 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 >> v1:172.21.15.87:6790/0 conn(0x7fa8b00a98f0 legacy=0x7fa8b01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 shutdown_connections 2024-03-25T18:25:05.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 >> 172.21.15.87:0/3933747014 conn(0x7fa8b00b7ea0 msgr2=0x7fa8b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:05.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 shutdown_connections 2024-03-25T18:25:05.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 -- 172.21.15.87:0/3933747014 wait complete. 2024-03-25T18:25:05.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:05.102+0000 7fa8b7fd0640 1 librados: shutdown 2024-03-25T18:25:05.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:11.922 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.033+0000 7f032e063640 10 monclient: get_monmap_and_config 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.033+0000 7f032e063640 10 monclient: build_initial_monmap 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.033+0000 7f032e063640 10 monclient: monmap: 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:12.034599+0000 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:12.034599+0000 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:12.035 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:12.036 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.034+0000 7f032e063640 1 Processor -- start 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.034+0000 7f032e063640 1 -- start start 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.034+0000 7f032e063640 10 monclient: init 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient(hunting): picked mon.noname-a con 0x7f03280a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient(hunting): picked mon.noname-b con 0x7f032813f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient(hunting): picked mon.noname-c con 0x7f03281430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328142520 con 0x7f03280a98f0 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280b28a0 con 0x7f03281430c0 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280b2a30 con 0x7f032813f710 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:12.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.035+0000 7f032e063640 10 monclient(hunting): authenticate will time out at 8427.330644s 2024-03-25T18:25:12.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.040+0000 7f0326ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f032813f710 0x7f03281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:37606/0 (socket says 172.21.15.87:37606) 2024-03-25T18:25:12.041 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.040+0000 7f0326ffd640 1 -- 172.21.15.87:0/1666115023 learned_addr learned my addr 172.21.15.87:0/1666115023 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:12.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312485180 0 0) 0x7f03280b2a30 con 0x7f032813f710 2024-03-25T18:25:12.051 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:25:12.052 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.050+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f032813f710 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257165720 0 0) 0x7f03280b28a0 con 0x7f03281430c0 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:25:12.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.054+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03280b2a30 con 0x7f03281430c0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833568683 0 0) 0x7f03280b2a30 con 0x7f03281430c0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 >> v1:172.21.15.87:6790/0 conn(0x7f032813f710 legacy=0x7f03281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 >> v1:172.21.15.87:6789/0 conn(0x7f03280a98f0 legacy=0x7f03280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280b44c0 con 0x7f03281430c0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:12.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f03200019f0 con 0x7f03281430c0 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f032e063640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f032e063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f03267fc640 10 monclient: dump: 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:12.068 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:12.069 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:12.069 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.066+0000 7f032e063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0320001b50 con 0x7f03281430c0 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 1 -- 172.21.15.87:0/1666115023 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0320001f20 con 0x7f03281430c0 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f032e063640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f032e063640 4 set_mon_vals no callback set 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.083 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.082+0000 7f03267fc640 10 monclient: dump: 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:12.084 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 10 monclient: shutdown 2024-03-25T18:25:12.085 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 1 -- 172.21.15.87:0/1666115023 >> v1:172.21.15.171:6789/0 conn(0x7f03281430c0 legacy=0x7f032814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 1 -- 172.21.15.87:0/1666115023 shutdown_connections 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 1 -- 172.21.15.87:0/1666115023 >> 172.21.15.87:0/1666115023 conn(0x7f03280b7ea0 msgr2=0x7f03280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 1 -- 172.21.15.87:0/1666115023 shutdown_connections 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.083+0000 7f032e063640 1 -- 172.21.15.87:0/1666115023 wait complete. 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.084+0000 7f032e063640 10 monclient: build_initial_monmap 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.084+0000 7f032e063640 10 monclient: monmap: 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:12.086 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.084+0000 7f032e063640 1 librados: starting msgr at 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.084+0000 7f032e063640 1 librados: starting objecter 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.084+0000 7f032e063640 1 Processor -- start 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 -- start start 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 librados: setting wanted keys 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 librados: calling monclient init 2024-03-25T18:25:12.087 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient: init 2024-03-25T18:25:12.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:12.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:12.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient(hunting): picked mon.noname-c con 0x7f03280a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:12.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient(hunting): picked mon.noname-a con 0x7f032813f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:12.088 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient(hunting): picked mon.noname-b con 0x7f03281430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:12.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328138330 con 0x7f032813f710 2024-03-25T18:25:12.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03281398c0 con 0x7f03281430c0 2024-03-25T18:25:12.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03281385f0 con 0x7f03280a98f0 2024-03-25T18:25:12.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:12.089 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.085+0000 7f032e063640 10 monclient(hunting): authenticate will time out at 8427.380752s 2024-03-25T18:25:12.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.092+0000 7f03277fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f03280a98f0 0x7f0328137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:37638/0 (socket says 172.21.15.87:37638) 2024-03-25T18:25:12.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.092+0000 7f0327fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f03281430c0 0x7f03281e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:43668/0 (socket says 172.21.15.87:43668) 2024-03-25T18:25:12.093 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.092+0000 7f03277fe640 1 -- 172.21.15.87:0/764028648 learned_addr learned my addr 172.21.15.87:0/764028648 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:12.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.095+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f03280a98f0 2024-03-25T18:25:12.096 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.096+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.098+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f03281430c0 2024-03-25T18:25:12.099 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.098+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060376699 0 0) 0x7f03281385f0 con 0x7f03280a98f0 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:25:12.104 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.103+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314004eb0 con 0x7f03280a98f0 2024-03-25T18:25:12.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.105+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f032813f710 2024-03-25T18:25:12.105 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.105+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518248646 0 0) 0x7f03281398c0 con 0x7f03281430c0 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:25:12.114 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.113+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03281385f0 con 0x7f03281430c0 2024-03-25T18:25:12.117 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018741428 0 0) 0x7f0314004eb0 con 0x7f03280a98f0 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03281398c0 con 0x7f03280a98f0 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f03180019d0 con 0x7f03280a98f0 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:12.118 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:12.119 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.119 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.116+0000 7f030ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:12.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.123+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857275192 0 0) 0x7f03281385f0 con 0x7f03281430c0 2024-03-25T18:25:12.124 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.123+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0314004eb0 con 0x7f03281430c0 2024-03-25T18:25:12.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.127+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f0320001f00 con 0x7f03281430c0 2024-03-25T18:25:12.127 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.127+0000 7f030ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.127+0000 7f030ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.127+0000 7f030ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:12.128 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.139 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2627167758 0 0) 0x7f03281398c0 con 0x7f03280a98f0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 >> v1:172.21.15.171:6789/0 conn(0x7f03281430c0 legacy=0x7f03281e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 >> v1:172.21.15.87:6789/0 conn(0x7f032813f710 legacy=0x7f03281d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 1 monclient: found mon.c 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328138b10 con 0x7f03280a98f0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f030ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 10 monclient: _renew_subs 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:12.140 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03281d7c10 con 0x7f03280a98f0 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 10 monclient: _renew_subs 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f032814e940 con 0x7f03280a98f0 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.138+0000 7f032e063640 1 librados: init done 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.139+0000 7f032e063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:12.141 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.139+0000 7f032e063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:12.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.139+0000 7f032e063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:12.142 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.139+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0328138780 con 0x7f03280a98f0 2024-03-25T18:25:12.430 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0318003610 con 0x7f03280a98f0 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f03180039a0 con 0x7f03280a98f0 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f030ffff640 10 monclient: dump: 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:12.431 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.429+0000 7f0325ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:12.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.430+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f0318003bc0 con 0x7f03280a98f0 2024-03-25T18:25:12.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.430+0000 7f0326ffd640 10 monclient: get_auth_request con 0x7f031406de20 auth_method 0 2024-03-25T18:25:12.432 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.431+0000 7f030ffff640 10 client.4703.objecter ms_handle_connect 0x7f031406de20 2024-03-25T18:25:12.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f03180547a0 con 0x7f03280a98f0 2024-03-25T18:25:12.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 10 client.4703.objecter ms_dispatch 0x7f0328000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:12.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:12.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:12.436 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T18:25:12.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03180502b0 con 0x7f03280a98f0 2024-03-25T18:25:12.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:12.437 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.435+0000 7f030ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:12.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.689+0000 7f032e063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:12.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.689+0000 7f032e063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:12.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.689+0000 7f032e063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:12.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.689+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0328138ca0 con 0x7f03280a98f0 2024-03-25T18:25:12.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.712+0000 7f030ffff640 1 -- 172.21.15.87:0/764028648 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f03180521e0 con 0x7f03280a98f0 2024-03-25T18:25:12.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.712+0000 7f030ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:12.713 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.712+0000 7f030ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:12.713 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:12.713 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.714+0000 7f032e063640 10 librados: watch_flush enter 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.714+0000 7f032e063640 10 librados: watch_flush exit 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.714+0000 7f032e063640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.714+0000 7f032e063640 10 client.4703.objecter successfully canceled tick 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 >> v1:172.21.15.171:6800/3417775508 conn(0x7f031406de20 legacy=0x7f03140702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 10 monclient: shutdown 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 >> v1:172.21.15.87:6790/0 conn(0x7f03280a98f0 legacy=0x7f0328137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 shutdown_connections 2024-03-25T18:25:12.716 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 >> 172.21.15.87:0/764028648 conn(0x7f03280b7ea0 msgr2=0x7f0328074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:12.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 shutdown_connections 2024-03-25T18:25:12.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 -- 172.21.15.87:0/764028648 wait complete. 2024-03-25T18:25:12.717 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:12.715+0000 7f032e063640 1 librados: shutdown 2024-03-25T18:25:12.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:13.380 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:01.094 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.430 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:01.143 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.470 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:01.183 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.482 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:01.195 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.486 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:01.165 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.492 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:01.171 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.495 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:01.208 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.523 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:01.236 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:13.525 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:01.203 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.534 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:19.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.647+0000 7f5e56add640 10 monclient: get_monmap_and_config 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.647+0000 7f5e56add640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.647+0000 7f5e56add640 10 monclient: monmap: 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:19.648530+0000 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:19.648530+0000 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:19.649 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.648+0000 7f5e56add640 1 Processor -- start 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.648+0000 7f5e56add640 1 -- start start 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.648+0000 7f5e56add640 10 monclient: init 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-c con 0x7f5e500a98f0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-b con 0x7f5e5013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-a con 0x7f5e501430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e50142520 con 0x7f5e501430c0 2024-03-25T18:25:19.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e500b28a0 con 0x7f5e500a98f0 2024-03-25T18:25:19.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e500b2a30 con 0x7f5e5013f710 2024-03-25T18:25:19.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.649+0000 7f5e56add640 10 monclient(hunting): authenticate will time out at 8434.944308s 2024-03-25T18:25:19.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.650+0000 7f5e54879640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5e500a98f0 0x7f5e500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:34138/0 (socket says 172.21.15.87:34138) 2024-03-25T18:25:19.652 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.650+0000 7f5e54879640 1 -- 172.21.15.87:0/3431282453 learned_addr learned my addr 172.21.15.87:0/3431282453 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047120814 0 0) 0x7f5e500b28a0 con 0x7f5e500a98f0 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:25:19.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.660+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e38003520 con 0x7f5e500a98f0 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510783345 0 0) 0x7f5e50142520 con 0x7f5e501430c0 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:25:19.667 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.666+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e500b28a0 con 0x7f5e501430c0 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437783407 0 0) 0x7f5e500b2a30 con 0x7f5e5013f710 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:25:19.673 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.671+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e50142520 con 0x7f5e5013f710 2024-03-25T18:25:19.681 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846526303 0 0) 0x7f5e500b28a0 con 0x7f5e501430c0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 >> v1:172.21.15.87:6790/0 conn(0x7f5e5013f710 legacy=0x7f5e501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 >> v1:172.21.15.171:6789/0 conn(0x7f5e500a98f0 legacy=0x7f5e500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500b44c0 con 0x7f5e501430c0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5e480019f0 con 0x7f5e501430c0 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e56add640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:25:19.682 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e56add640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e477fe640 10 monclient: dump: 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:19.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.680+0000 7f5e56add640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e48001b50 con 0x7f5e501430c0 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 1 -- 172.21.15.87:0/3431282453 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5e48001f60 con 0x7f5e501430c0 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e477fe640 10 monclient: dump: 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:19.702 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e56add640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:19.703 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.701+0000 7f5e56add640 4 set_mon_vals no callback set 2024-03-25T18:25:19.704 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 10 monclient: shutdown 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 1 -- 172.21.15.87:0/3431282453 >> v1:172.21.15.87:6789/0 conn(0x7f5e501430c0 legacy=0x7f5e5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 1 -- 172.21.15.87:0/3431282453 shutdown_connections 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 1 -- 172.21.15.87:0/3431282453 >> 172.21.15.87:0/3431282453 conn(0x7f5e500b7ea0 msgr2=0x7f5e500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 1 -- 172.21.15.87:0/3431282453 shutdown_connections 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.702+0000 7f5e56add640 1 -- 172.21.15.87:0/3431282453 wait complete. 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.703+0000 7f5e56add640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.703+0000 7f5e56add640 10 monclient: monmap: 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:19.705 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.703+0000 7f5e56add640 1 librados: starting msgr at 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.703+0000 7f5e56add640 1 librados: starting objecter 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.703+0000 7f5e56add640 1 Processor -- start 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.704+0000 7f5e56add640 1 -- start start 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.704+0000 7f5e56add640 1 librados: setting wanted keys 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.704+0000 7f5e56add640 1 librados: calling monclient init 2024-03-25T18:25:19.706 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.704+0000 7f5e56add640 10 monclient: init 2024-03-25T18:25:19.707 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.704+0000 7f5e56add640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-c con 0x7f5e500a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-b con 0x7f5e5013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient(hunting): picked mon.noname-a con 0x7f5e501430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e501d48f0 con 0x7f5e501430c0 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e501397d0 con 0x7f5e5013f710 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e501d4b50 con 0x7f5e500a98f0 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.708 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.705+0000 7f5e56add640 10 monclient(hunting): authenticate will time out at 8434.999990s 2024-03-25T18:25:19.725 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.725+0000 7f5e47fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5e5013f710 0x7f5e500aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:34176/0 (socket says 172.21.15.87:34176) 2024-03-25T18:25:19.726 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.725+0000 7f5e47fff640 1 -- 172.21.15.87:0/4000234725 learned_addr learned my addr 172.21.15.87:0/4000234725 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:19.728 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.728+0000 7f5e457fa640 10 client.?.objecter ms_handle_connect 0x7f5e5013f710 2024-03-25T18:25:19.729 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.728+0000 7f5e457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.729+0000 7f5e457fa640 10 client.?.objecter ms_handle_connect 0x7f5e501430c0 2024-03-25T18:25:19.730 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.729+0000 7f5e457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.734+0000 7f5e457fa640 10 client.?.objecter ms_handle_connect 0x7f5e500a98f0 2024-03-25T18:25:19.735 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.734+0000 7f5e457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774999645 0 0) 0x7f5e501397d0 con 0x7f5e5013f710 2024-03-25T18:25:19.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.743 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:25:19.744 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.742+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c004eb0 con 0x7f5e5013f710 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439480809 0 0) 0x7f5e501d48f0 con 0x7f5e501430c0 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:25:19.750 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.749+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e501397d0 con 0x7f5e501430c0 2024-03-25T18:25:19.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187521412 0 0) 0x7f5e501d4b50 con 0x7f5e500a98f0 2024-03-25T18:25:19.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.753 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:25:19.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.752+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e501d48f0 con 0x7f5e500a98f0 2024-03-25T18:25:19.754 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670243995 0 0) 0x7f5e3c004eb0 con 0x7f5e5013f710 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e501d4b50 con 0x7f5e5013f710 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5e40001a10 con 0x7f5e5013f710 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 10 monclient(hunting): dump: 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:19.755 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:19.756 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:19.756 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.753+0000 7f5e457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.761+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077442926 0 0) 0x7f5e501d4b50 con 0x7f5e5013f710 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 >> v1:172.21.15.87:6790/0 conn(0x7f5e500a98f0 legacy=0x7f5e501d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 >> v1:172.21.15.87:6789/0 conn(0x7f5e501430c0 legacy=0x7f5e501d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 1 monclient: found mon.b 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.763 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500b16b0 con 0x7f5e5013f710 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 10 monclient: _renew_subs 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e500b0c10 con 0x7f5e5013f710 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 10 monclient: _renew_subs 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e500b0610 con 0x7f5e5013f710 2024-03-25T18:25:19.764 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.762+0000 7f5e56add640 1 librados: init done 2024-03-25T18:25:19.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.763+0000 7f5e56add640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.763+0000 7f5e56add640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.763+0000 7f5e56add640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:19.765 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:19.763+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e501d4ce0 con 0x7f5e5013f710 2024-03-25T18:25:20.042 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e40001c70 con 0x7f5e5013f710 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f5e40003a60 con 0x7f5e5013f710 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e457fa640 10 monclient: dump: 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:20.043 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:20.044 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:20.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.041+0000 7f5e46ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:20.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.042+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f5e40003c80 con 0x7f5e5013f710 2024-03-25T18:25:20.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.043+0000 7f5e54879640 10 monclient: get_auth_request con 0x7f5e3c06ddf0 auth_method 0 2024-03-25T18:25:20.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.043+0000 7f5e457fa640 10 client.4705.objecter ms_handle_connect 0x7f5e3c06ddf0 2024-03-25T18:25:20.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f5e400548b0 con 0x7f5e5013f710 2024-03-25T18:25:20.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 10 client.4705.objecter ms_dispatch 0x7f5e50000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:20.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:20.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:20.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:25:20.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e400503c0 con 0x7f5e5013f710 2024-03-25T18:25:20.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.052+0000 7f5e457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:20.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.280+0000 7f5e56add640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.280+0000 7f5e56add640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.280+0000 7f5e56add640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:20.281 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.280+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e500627b0 con 0x7f5e5013f710 2024-03-25T18:25:20.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.311+0000 7f5e457fa640 1 -- 172.21.15.87:0/4000234725 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f5e400522f0 con 0x7f5e5013f710 2024-03-25T18:25:20.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.311+0000 7f5e457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.312 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.311+0000 7f5e457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:20.312 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:20.312 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:20.314 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.313+0000 7f5e56add640 10 librados: watch_flush enter 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 10 librados: watch_flush exit 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 >> v1:172.21.15.171:6800/3417775508 conn(0x7f5e3c06ddf0 legacy=0x7f5e3c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 10 monclient: shutdown 2024-03-25T18:25:20.315 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 >> v1:172.21.15.171:6789/0 conn(0x7f5e5013f710 legacy=0x7f5e500aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 shutdown_connections 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 >> 172.21.15.87:0/4000234725 conn(0x7f5e500b7ea0 msgr2=0x7f5e50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 shutdown_connections 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 -- 172.21.15.87:0/4000234725 wait complete. 2024-03-25T18:25:20.316 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:20.314+0000 7f5e56add640 1 librados: shutdown 2024-03-25T18:25:20.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:27.134 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.245+0000 7fca2a9ba640 10 monclient: get_monmap_and_config 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.245+0000 7fca2a9ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.245+0000 7fca2a9ba640 10 monclient: monmap: 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:27.247006+0000 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:27.247006+0000 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:27.247 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:27.248 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:27.248 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.245+0000 7fca2a9ba640 1 Processor -- start 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.246+0000 7fca2a9ba640 1 -- start start 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.246+0000 7fca2a9ba640 10 monclient: init 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:27.249 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fca240a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fca2413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fca241430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca24142520 con 0x7fca240a98f0 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca240b28a0 con 0x7fca241430c0 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca240b2a30 con 0x7fca2413f710 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:27.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.247+0000 7fca2a9ba640 10 monclient(hunting): authenticate will time out at 8442.543275s 2024-03-25T18:25:27.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.255+0000 7fca237fe640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fca2413f710 0x7fca241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:45404/0 (socket says 172.21.15.87:45404) 2024-03-25T18:25:27.257 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.255+0000 7fca237fe640 1 -- 172.21.15.87:0/2906616145 learned_addr learned my addr 172.21.15.87:0/2906616145 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953068499 0 0) 0x7fca240b2a30 con 0x7fca2413f710 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:25:27.271 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.269+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c003520 con 0x7fca2413f710 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139950459 0 0) 0x7fca240b28a0 con 0x7fca241430c0 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:25:27.279 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.277+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca240b2a30 con 0x7fca241430c0 2024-03-25T18:25:27.287 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499958241 0 0) 0x7fca0c003520 con 0x7fca2413f710 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 >> v1:172.21.15.171:6789/0 conn(0x7fca241430c0 legacy=0x7fca2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 >> v1:172.21.15.87:6789/0 conn(0x7fca240a98f0 legacy=0x7fca240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 monclient: found mon.noname-b 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca240b44c0 con 0x7fca2413f710 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fca18001690 con 0x7fca2413f710 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca22ffd640 10 monclient: dump: 2024-03-25T18:25:27.288 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca2a9ba640 5 monclient: authenticate success, global_id 4712 2024-03-25T18:25:27.289 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.285+0000 7fca2a9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca180017f0 con 0x7fca2413f710 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 1 -- 172.21.15.87:0/2906616145 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fca18001c00 con 0x7fca2413f710 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca2a9ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca2a9ba640 4 set_mon_vals no callback set 2024-03-25T18:25:27.303 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.301+0000 7fca22ffd640 10 monclient: dump: 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:27.304 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 10 monclient: shutdown 2024-03-25T18:25:27.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:27.306 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 1 -- 172.21.15.87:0/2906616145 >> v1:172.21.15.87:6790/0 conn(0x7fca2413f710 legacy=0x7fca241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 1 -- 172.21.15.87:0/2906616145 shutdown_connections 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 1 -- 172.21.15.87:0/2906616145 >> 172.21.15.87:0/2906616145 conn(0x7fca240b7ea0 msgr2=0x7fca240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 1 -- 172.21.15.87:0/2906616145 shutdown_connections 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.302+0000 7fca2a9ba640 1 -- 172.21.15.87:0/2906616145 wait complete. 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.303+0000 7fca2a9ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.303+0000 7fca2a9ba640 10 monclient: monmap: 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:27.307 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.303+0000 7fca2a9ba640 1 librados: starting msgr at 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.303+0000 7fca2a9ba640 1 librados: starting objecter 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.303+0000 7fca2a9ba640 1 Processor -- start 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 -- start start 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 librados: setting wanted keys 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 librados: calling monclient init 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient: init 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fca240a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fca2413f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fca241430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca24137df0 con 0x7fca240a98f0 2024-03-25T18:25:27.308 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca241337a0 con 0x7fca241430c0 2024-03-25T18:25:27.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca24133930 con 0x7fca2413f710 2024-03-25T18:25:27.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:27.309 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.304+0000 7fca2a9ba640 10 monclient(hunting): authenticate will time out at 8442.600751s 2024-03-25T18:25:27.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.315+0000 7fca23fff640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7fca240a98f0 0x7fca241392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:58974/0 (socket says 172.21.15.87:58974) 2024-03-25T18:25:27.317 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.315+0000 7fca23fff640 1 -- 172.21.15.87:0/968027996 learned_addr learned my addr 172.21.15.87:0/968027996 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:27.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.325+0000 7fca20ff9640 10 client.?.objecter ms_handle_connect 0x7fca241430c0 2024-03-25T18:25:27.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.325+0000 7fca20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.325+0000 7fca20ff9640 10 client.?.objecter ms_handle_connect 0x7fca2413f710 2024-03-25T18:25:27.327 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.325+0000 7fca20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.331+0000 7fca20ff9640 10 client.?.objecter ms_handle_connect 0x7fca240a98f0 2024-03-25T18:25:27.333 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.331+0000 7fca20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492296549 0 0) 0x7fca24133930 con 0x7fca2413f710 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:25:27.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.341+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca10004eb0 con 0x7fca2413f710 2024-03-25T18:25:27.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788858891 0 0) 0x7fca241337a0 con 0x7fca241430c0 2024-03-25T18:25:27.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.353 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca24133930 con 0x7fca241430c0 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698940607 0 0) 0x7fca24137df0 con 0x7fca240a98f0 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:25:27.354 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.351+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca241337a0 con 0x7fca240a98f0 2024-03-25T18:25:27.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.356+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2424286628 0 0) 0x7fca10004eb0 con 0x7fca2413f710 2024-03-25T18:25:27.358 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.356+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca24137df0 con 0x7fca2413f710 2024-03-25T18:25:27.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3296734644 0 0) 0x7fca24133930 con 0x7fca241430c0 2024-03-25T18:25:27.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca10004eb0 con 0x7fca241430c0 2024-03-25T18:25:27.367 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fca1c001b50 con 0x7fca241430c0 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.368 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.365+0000 7fca20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:27.371 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fca18001b80 con 0x7fca2413f710 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4225439690 0 0) 0x7fca24137df0 con 0x7fca2413f710 2024-03-25T18:25:27.372 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 >> v1:172.21.15.171:6789/0 conn(0x7fca241430c0 legacy=0x7fca24133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 >> v1:172.21.15.87:6789/0 conn(0x7fca240a98f0 legacy=0x7fca241392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 monclient: found mon.c 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca241367d0 con 0x7fca2413f710 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 10 monclient: _renew_subs 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca241369e0 con 0x7fca2413f710 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 10 monclient: _renew_subs 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.373 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2413b640 con 0x7fca2413f710 2024-03-25T18:25:27.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.369+0000 7fca2a9ba640 1 librados: init done 2024-03-25T18:25:27.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.370+0000 7fca2a9ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:27.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.370+0000 7fca2a9ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:27.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.370+0000 7fca2a9ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.374 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.370+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca240627b0 con 0x7fca2413f710 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca18004a90 con 0x7fca2413f710 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fca18004da0 con 0x7fca2413f710 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca20ff9640 10 monclient: dump: 2024-03-25T18:25:27.683 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:27.684 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.681+0000 7fca227fc640 4 set_mon_vals no callback set 2024-03-25T18:25:27.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.682+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fca1801ad70 con 0x7fca2413f710 2024-03-25T18:25:27.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.682+0000 7fca23fff640 10 monclient: get_auth_request con 0x7fca1006dfc0 auth_method 0 2024-03-25T18:25:27.685 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.682+0000 7fca20ff9640 10 client.4715.objecter ms_handle_connect 0x7fca1006dfc0 2024-03-25T18:25:27.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7fca18054840 con 0x7fca2413f710 2024-03-25T18:25:27.689 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 10 client.4715.objecter ms_dispatch 0x7fca24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca18050350 con 0x7fca2413f710 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:27.690 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.687+0000 7fca20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.924+0000 7fca2a9ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.924+0000 7fca2a9ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.924+0000 7fca2a9ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.924+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca24074cc0 con 0x7fca2413f710 2024-03-25T18:25:27.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.959+0000 7fca20ff9640 1 -- 172.21.15.87:0/968027996 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7fca18052280 con 0x7fca2413f710 2024-03-25T18:25:27.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.959+0000 7fca20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:27.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.959+0000 7fca20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:27.961 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:27.961 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:27.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 10 librados: watch_flush enter 2024-03-25T18:25:27.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 10 librados: watch_flush exit 2024-03-25T18:25:27.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T18:25:27.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 10 client.4715.objecter successfully canceled tick 2024-03-25T18:25:27.963 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 >> v1:172.21.15.171:6800/3417775508 conn(0x7fca1006dfc0 legacy=0x7fca10070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 10 monclient: shutdown 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 >> v1:172.21.15.87:6790/0 conn(0x7fca2413f710 legacy=0x7fca241376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 shutdown_connections 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 >> 172.21.15.87:0/968027996 conn(0x7fca240b7ea0 msgr2=0x7fca24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 shutdown_connections 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 -- 172.21.15.87:0/968027996 wait complete. 2024-03-25T18:25:27.964 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:27.961+0000 7fca2a9ba640 1 librados: shutdown 2024-03-25T18:25:27.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:33.442 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:21.154 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.484 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:21.162 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.492 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:21.205 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.498 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:21.176 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.517 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:21.231 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.539 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:21.218 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.575 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:21.288 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.578 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:21.291 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:33.604 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:21.317 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:34.781 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.972+0000 7f4e61f48640 10 monclient: get_monmap_and_config 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.972+0000 7f4e61f48640 10 monclient: build_initial_monmap 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.972+0000 7f4e61f48640 10 monclient: monmap: 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:34.973364+0000 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:34.973364+0000 2024-03-25T18:25:34.973 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.973+0000 7f4e61f48640 1 Processor -- start 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.973+0000 7f4e61f48640 1 -- start start 2024-03-25T18:25:34.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.973+0000 7f4e61f48640 10 monclient: init 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c06adf0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c0aee20 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c0afe80 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0729c0 con 0x7f4e5c0afe80 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c143e60 con 0x7f4e5c06adf0 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c13db40 con 0x7f4e5c0aee20 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.974+0000 7f4e61f48640 10 monclient(hunting): authenticate will time out at 8450.269306s 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.975+0000 7f4e5affd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f4e5c0aee20 0x7f4e5c13d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:52708/0 (socket says 172.21.15.87:52708) 2024-03-25T18:25:34.976 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.975+0000 7f4e5affd640 1 -- 172.21.15.87:0/1464803166 learned_addr learned my addr 172.21.15.87:0/1464803166 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:34.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840280955 0 0) 0x7f4e5c13db40 con 0x7f4e5c0aee20 2024-03-25T18:25:34.988 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.989 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.989 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.989 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:25:34.989 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.987+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e44003520 con 0x7f4e5c0aee20 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333038715 0 0) 0x7f4e5c143e60 con 0x7f4e5c06adf0 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:25:34.998 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:34.997+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c13db40 con 0x7f4e5c06adf0 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1154536825 0 0) 0x7f4e44003520 con 0x7f4e5c0aee20 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 >> v1:172.21.15.171:6789/0 conn(0x7f4e5c06adf0 legacy=0x7f4e5c143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 >> v1:172.21.15.87:6789/0 conn(0x7f4e5c0afe80 legacy=0x7f4e5c14baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c13dcd0 con 0x7f4e5c0aee20 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:35.006 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e50001690 con 0x7f4e5c0aee20 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e5a7fc640 10 monclient: dump: 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:35.007 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:35.008 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e61f48640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:25:35.008 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.005+0000 7f4e61f48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e500017f0 con 0x7f4e5c0aee20 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 1 -- 172.21.15.87:0/1464803166 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e50001c00 con 0x7f4e5c0aee20 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e61f48640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e61f48640 4 set_mon_vals no callback set 2024-03-25T18:25:35.013 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.012+0000 7f4e5a7fc640 10 monclient: dump: 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:35.014 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:35.016 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 10 monclient: shutdown 2024-03-25T18:25:35.016 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:35.016 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 1 -- 172.21.15.87:0/1464803166 >> v1:172.21.15.87:6790/0 conn(0x7f4e5c0aee20 legacy=0x7f4e5c13d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 1 -- 172.21.15.87:0/1464803166 shutdown_connections 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 1 -- 172.21.15.87:0/1464803166 >> 172.21.15.87:0/1464803166 conn(0x7f4e5c0b4a80 msgr2=0x7f4e5c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 1 -- 172.21.15.87:0/1464803166 shutdown_connections 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.013+0000 7f4e61f48640 1 -- 172.21.15.87:0/1464803166 wait complete. 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.014+0000 7f4e61f48640 10 monclient: build_initial_monmap 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.014+0000 7f4e61f48640 10 monclient: monmap: 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:35.017 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.014+0000 7f4e61f48640 1 librados: starting msgr at 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.014+0000 7f4e61f48640 1 librados: starting objecter 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.014+0000 7f4e61f48640 1 Processor -- start 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 -- start start 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 librados: setting wanted keys 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 librados: calling monclient init 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient: init 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c06adf0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c0afe80 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c0ae580 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c1399c0 con 0x7f4e5c06adf0 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0adc10 con 0x7f4e5c0ae580 2024-03-25T18:25:35.018 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0ade80 con 0x7f4e5c0afe80 2024-03-25T18:25:35.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:35.019 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.015+0000 7f4e61f48640 10 monclient(hunting): authenticate will time out at 8450.310642s 2024-03-25T18:25:35.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.026+0000 7f4e5bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4e5c0ae580 0x7f4e5c14b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:33232/0 (socket says 172.21.15.87:33232) 2024-03-25T18:25:35.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.026+0000 7f4e5bfff640 1 -- 172.21.15.87:0/1999132854 learned_addr learned my addr 172.21.15.87:0/1999132854 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:35.027 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.027+0000 7f4e43fff640 10 client.?.objecter ms_handle_connect 0x7f4e5c0ae580 2024-03-25T18:25:35.028 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.027+0000 7f4e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.034+0000 7f4e43fff640 10 client.?.objecter ms_handle_connect 0x7f4e5c06adf0 2024-03-25T18:25:35.035 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.035+0000 7f4e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.037+0000 7f4e43fff640 10 client.?.objecter ms_handle_connect 0x7f4e5c0afe80 2024-03-25T18:25:35.037 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.037+0000 7f4e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.044 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300654544 0 0) 0x7f4e5c0adc10 con 0x7f4e5c0ae580 2024-03-25T18:25:35.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:25:35.045 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.044+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48004eb0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.053 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175023713 0 0) 0x7f4e5c0ade80 con 0x7f4e5c0afe80 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c0adc10 con 0x7f4e5c0afe80 2024-03-25T18:25:35.054 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195046372 0 0) 0x7f4e5c13db40 con 0x7f4e5c06adf0 2024-03-25T18:25:35.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:25:35.055 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.053+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c0ade80 con 0x7f4e5c06adf0 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869886791 0 0) 0x7f4e48004eb0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5c13db40 con 0x7f4e5c0ae580 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e54001c90 con 0x7f4e5c0ae580 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 10 monclient(hunting): dump: 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:35.059 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:35.060 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.058+0000 7f4e43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:35.067 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4243940315 0 0) 0x7f4e5c13db40 con 0x7f4e5c0ae580 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 >> v1:172.21.15.87:6790/0 conn(0x7f4e5c0afe80 legacy=0x7f4e5c0abb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 >> v1:172.21.15.87:6789/0 conn(0x7f4e5c06adf0 legacy=0x7f4e5c1d8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 1 monclient: found mon.b 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c0ad000 con 0x7f4e5c0ae580 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e61f48640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e61f48640 10 monclient: _renew_subs 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e61f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:35.068 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.066+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c0acc20 con 0x7f4e5c0ae580 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.067+0000 7f4e61f48640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.067+0000 7f4e61f48640 10 monclient: _renew_subs 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.067+0000 7f4e61f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.067+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c1d9580 con 0x7f4e5c0ae580 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.067+0000 7f4e61f48640 1 librados: init done 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.068+0000 7f4e61f48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.068+0000 7f4e61f48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.069 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.068+0000 7f4e61f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:35.070 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.068+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5c0ace60 con 0x7f4e5c0ae580 2024-03-25T18:25:35.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.070+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e54003990 con 0x7f4e5c0ae580 2024-03-25T18:25:35.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.070+0000 7f4e43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:35.071 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.070+0000 7f4e59ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.342+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4e54003bc0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.342+0000 7f4e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.342+0000 7f4e43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.342+0000 7f4e43fff640 10 monclient: dump: 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:35.343 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:35.344 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:35.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.343+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4e54003da0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.343+0000 7f4e5b7fe640 10 monclient: get_auth_request con 0x7f4e4806dda0 auth_method 0 2024-03-25T18:25:35.345 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.344+0000 7f4e43fff640 10 client.4717.objecter ms_handle_connect 0x7f4e4806dda0 2024-03-25T18:25:35.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.345+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f4e540549d0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.345+0000 7f4e43fff640 10 client.4717.objecter ms_dispatch 0x7f4e5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:35.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.345+0000 7f4e43fff640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:35.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.345+0000 7f4e43fff640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:35.346 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.345+0000 7f4e43fff640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:25:35.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.349+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e540504e0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.349+0000 7f4e43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.350 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.349+0000 7f4e43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:35.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.605+0000 7f4e61f48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:35.606 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.605+0000 7f4e61f48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:35.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.605+0000 7f4e61f48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:25:35.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.605+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e5c0627b0 con 0x7f4e5c0ae580 2024-03-25T18:25:35.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.631+0000 7f4e43fff640 1 -- 172.21.15.87:0/1999132854 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f4e54052410 con 0x7f4e5c0ae580 2024-03-25T18:25:35.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.631+0000 7f4e43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:35.632 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.631+0000 7f4e43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:35.632 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:35.632 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 10 librados: watch_flush enter 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 10 librados: watch_flush exit 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4e4806dda0 legacy=0x7f4e48070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 10 monclient: shutdown 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 >> v1:172.21.15.171:6789/0 conn(0x7f4e5c0ae580 legacy=0x7f4e5c14b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 shutdown_connections 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 >> 172.21.15.87:0/1999132854 conn(0x7f4e5c0b4a80 msgr2=0x7f4e5c13d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 shutdown_connections 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.633+0000 7f4e61f48640 1 -- 172.21.15.87:0/1999132854 wait complete. 2024-03-25T18:25:35.635 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:35.634+0000 7f4e61f48640 1 librados: shutdown 2024-03-25T18:25:35.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:41.280 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:28.993 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.287 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:29.000 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.295 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:28.974 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.298 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:28.976 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.309 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:29.022 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.309 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:29.022 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.312 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:29.025 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.315 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:28.994 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:41.375 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:29.088 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:42.453 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:42.569 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.568+0000 7f2058f3e640 10 monclient: get_monmap_and_config 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.568+0000 7f2058f3e640 10 monclient: build_initial_monmap 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.568+0000 7f2058f3e640 10 monclient: monmap: 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:42.569568+0000 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:42.569568+0000 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:42.570 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.571 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.569+0000 7f2058f3e640 1 Processor -- start 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.569+0000 7f2058f3e640 1 -- start start 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.569+0000 7f2058f3e640 10 monclient: init 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-b con 0x7f20540a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-a con 0x7f205413f710 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f20541430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054142520 con 0x7f205413f710 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540b28a0 con 0x7f20541430c0 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540b2a30 con 0x7f20540a98f0 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:42.572 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.570+0000 7f2058f3e640 10 monclient(hunting): authenticate will time out at 8457.865192s 2024-03-25T18:25:42.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.574+0000 7f2052d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f20541430c0 0x7f205414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:50118/0 (socket says 172.21.15.87:50118) 2024-03-25T18:25:42.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.574+0000 7f2052d9d640 1 -- 172.21.15.87:0/296034199 learned_addr learned my addr 172.21.15.87:0/296034199 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469516905 0 0) 0x7f20540b28a0 con 0x7f20541430c0 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c003520 con 0x7f20541430c0 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107666001 0 0) 0x7f20540b2a30 con 0x7f20540a98f0 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.588 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:25:42.589 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.587+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540b28a0 con 0x7f20540a98f0 2024-03-25T18:25:42.600 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361794376 0 0) 0x7f2054142520 con 0x7f205413f710 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c004a30 con 0x7f205413f710 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285063158 0 0) 0x7f20540b28a0 con 0x7f20540a98f0 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 >> v1:172.21.15.171:6789/0 conn(0x7f20541430c0 legacy=0x7f205414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 >> v1:172.21.15.87:6789/0 conn(0x7f205413f710 legacy=0x7f20541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 monclient: found mon.noname-b 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.87:6790/0 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540b44c0 con 0x7f20540a98f0 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:42.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f20440015f0 con 0x7f20540a98f0 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f205159a640 10 monclient: dump: 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.602 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f2058f3e640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.599+0000 7f2058f3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:42.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.2 v1:172.21.15.87:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2044001750 con 0x7f20540a98f0 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 1 -- 172.21.15.87:0/296034199 <== mon.2 v1:172.21.15.87:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2044001ba0 con 0x7f20540a98f0 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f205159a640 10 monclient: dump: 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:42.604 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:42.605 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f2058f3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:42.605 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.602+0000 7f2058f3e640 4 set_mon_vals no callback set 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.603+0000 7f2058f3e640 10 monclient: shutdown 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.603+0000 7f2058f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.603+0000 7f2058f3e640 1 -- 172.21.15.87:0/296034199 >> v1:172.21.15.87:6790/0 conn(0x7f20540a98f0 legacy=0x7f20540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.603+0000 7f2058f3e640 1 -- 172.21.15.87:0/296034199 shutdown_connections 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.604+0000 7f2058f3e640 1 -- 172.21.15.87:0/296034199 >> 172.21.15.87:0/296034199 conn(0x7f20540b7ea0 msgr2=0x7f20540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.604+0000 7f2058f3e640 1 -- 172.21.15.87:0/296034199 shutdown_connections 2024-03-25T18:25:42.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.604+0000 7f2058f3e640 1 -- 172.21.15.87:0/296034199 wait complete. 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.604+0000 7f2058f3e640 10 monclient: build_initial_monmap 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.604+0000 7f2058f3e640 10 monclient: monmap: 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 librados: starting msgr at 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 librados: starting objecter 2024-03-25T18:25:42.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 Processor -- start 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 -- start start 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 librados: setting wanted keys 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 1 librados: calling monclient init 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.605+0000 7f2058f3e640 10 monclient: init 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f20540a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-b con 0x7f205413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient(hunting): picked mon.noname-a con 0x7f20541430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054135bb0 con 0x7f20541430c0 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054136ca0 con 0x7f205413f710 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054136f80 con 0x7f20540a98f0 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:42.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.606+0000 7f2058f3e640 10 monclient(hunting): authenticate will time out at 8457.901133s 2024-03-25T18:25:42.614 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.613+0000 7f205259c640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f20540a98f0 0x7f20540b6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:44168/0 (socket says 172.21.15.87:44168) 2024-03-25T18:25:42.614 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.613+0000 7f205259c640 1 -- 172.21.15.87:0/3303081048 learned_addr learned my addr 172.21.15.87:0/3303081048 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:42.614 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.613+0000 7f2042ffd640 10 client.?.objecter ms_handle_connect 0x7f20540a98f0 2024-03-25T18:25:42.614 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.613+0000 7f2042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:42.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.621+0000 7f2042ffd640 10 client.?.objecter ms_handle_connect 0x7f205413f710 2024-03-25T18:25:42.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.621+0000 7f2042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188901732 0 0) 0x7f2054136f80 con 0x7f20540a98f0 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:25:42.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.626+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034004eb0 con 0x7f20540a98f0 2024-03-25T18:25:42.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.628+0000 7f2042ffd640 10 client.?.objecter ms_handle_connect 0x7f20541430c0 2024-03-25T18:25:42.629 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.628+0000 7f2042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745936399 0 0) 0x7f2054136ca0 con 0x7f205413f710 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:25:42.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.632+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2054136f80 con 0x7f205413f710 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541471712 0 0) 0x7f2034004eb0 con 0x7f20540a98f0 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2054136ca0 con 0x7f20540a98f0 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2044001b60 con 0x7f20540a98f0 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:42.636 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.637 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.635+0000 7f2042ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245345742 0 0) 0x7f2054135bb0 con 0x7f20541430c0 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:25:42.639 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.638+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034004eb0 con 0x7f20541430c0 2024-03-25T18:25:42.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560715151 0 0) 0x7f2054136f80 con 0x7f205413f710 2024-03-25T18:25:42.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2054135bb0 con 0x7f205413f710 2024-03-25T18:25:42.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f20480018b0 con 0x7f205413f710 2024-03-25T18:25:42.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.642 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.641+0000 7f2042ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:42.643 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 121470696 0 0) 0x7f2054136ca0 con 0x7f20540a98f0 2024-03-25T18:25:42.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 >> v1:172.21.15.171:6789/0 conn(0x7f205413f710 legacy=0x7f20541352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.644 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 >> v1:172.21.15.87:6789/0 conn(0x7f20541430c0 legacy=0x7f2054137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 1 monclient: found mon.c 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054137260 con 0x7f20540a98f0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 10 monclient: _renew_subs 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2054137540 con 0x7f20540a98f0 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:42.645 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 10 monclient: _renew_subs 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20541e3690 con 0x7f20540a98f0 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.643+0000 7f2058f3e640 1 librados: init done 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.644+0000 7f2058f3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.644+0000 7f2058f3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:42.646 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.644+0000 7f2058f3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:42.647 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.644+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20540627b0 con 0x7f20540a98f0 2024-03-25T18:25:42.945 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20440038a0 con 0x7f20540a98f0 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2044001250 con 0x7f20540a98f0 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2042ffd640 10 monclient: dump: 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:42.946 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.944+0000 7f2050d99640 4 set_mon_vals no callback set 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.945+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2044003cf0 con 0x7f20540a98f0 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.946+0000 7f2051d9b640 10 monclient: get_auth_request con 0x7f203406dfb0 auth_method 0 2024-03-25T18:25:42.947 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.946+0000 7f2042ffd640 10 client.4727.objecter ms_handle_connect 0x7f203406dfb0 2024-03-25T18:25:42.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f2044054920 con 0x7f20540a98f0 2024-03-25T18:25:42.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 10 client.4727.objecter ms_dispatch 0x7f2054000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:42.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:42.952 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:42.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T18:25:42.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2044050260 con 0x7f20540a98f0 2024-03-25T18:25:42.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:42.953 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:42.951+0000 7f2042ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:43.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.202+0000 7f2058f3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.202+0000 7f2058f3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.202+0000 7f2058f3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.87:6790/0 2024-03-25T18:25:43.203 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.202+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 --> v1:172.21.15.87:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f205413de50 con 0x7f20540a98f0 2024-03-25T18:25:43.237 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.236+0000 7f2042ffd640 1 -- 172.21.15.87:0/3303081048 <== mon.2 v1:172.21.15.87:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f2044052360 con 0x7f20540a98f0 2024-03-25T18:25:43.237 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.236+0000 7f2042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.237 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.236+0000 7f2042ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:43.237 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:43.237 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.238+0000 7f2058f3e640 10 librados: watch_flush enter 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.238+0000 7f2058f3e640 10 librados: watch_flush exit 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.238+0000 7f2058f3e640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.238+0000 7f2058f3e640 10 client.4727.objecter successfully canceled tick 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 >> v1:172.21.15.171:6800/3417775508 conn(0x7f203406dfb0 legacy=0x7f2034070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 10 monclient: shutdown 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 >> v1:172.21.15.87:6790/0 conn(0x7f20540a98f0 legacy=0x7f20540b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 shutdown_connections 2024-03-25T18:25:43.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 >> 172.21.15.87:0/3303081048 conn(0x7f20540b7ea0 msgr2=0x7f2054074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:43.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 shutdown_connections 2024-03-25T18:25:43.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 -- 172.21.15.87:0/3303081048 wait complete. 2024-03-25T18:25:43.241 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:43.239+0000 7f2058f3e640 1 librados: shutdown 2024-03-25T18:25:43.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:50.058 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.171+0000 7f4ecbd4d640 10 monclient: get_monmap_and_config 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.171+0000 7f4ecbd4d640 10 monclient: build_initial_monmap 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.171+0000 7f4ecbd4d640 10 monclient: monmap: 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:50.172127+0000 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:50.172127+0000 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:50.172 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.171+0000 7f4ecbd4d640 1 Processor -- start 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 1 -- start start 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient: init 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec4142520 con 0x7f4ec41430c0 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40b28a0 con 0x7f4ec413f710 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40b2a30 con 0x7f4ec40a98f0 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:50.174 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.172+0000 7f4ecbd4d640 10 monclient(hunting): authenticate will time out at 8465.467822s 2024-03-25T18:25:50.176 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.176+0000 7f4ec92e8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4ec413f710 0x7f4ec41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:39866/0 (socket says 172.21.15.87:39866) 2024-03-25T18:25:50.177 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.176+0000 7f4ec92e8640 1 -- 172.21.15.87:0/869749216 learned_addr learned my addr 172.21.15.87:0/869749216 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636544829 0 0) 0x7f4ec40b28a0 con 0x7f4ec413f710 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:25:50.186 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.185+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0003520 con 0x7f4ec413f710 2024-03-25T18:25:50.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655653396 0 0) 0x7f4ec40b2a30 con 0x7f4ec40a98f0 2024-03-25T18:25:50.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.200 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:25:50.201 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.199+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40b28a0 con 0x7f4ec40a98f0 2024-03-25T18:25:50.205 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874565167 0 0) 0x7f4eb0003520 con 0x7f4ec413f710 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 >> v1:172.21.15.87:6790/0 conn(0x7f4ec40a98f0 legacy=0x7f4ec40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 >> v1:172.21.15.87:6789/0 conn(0x7f4ec41430c0 legacy=0x7f4ec414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: _finish_hunting 0 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 monclient: found mon.noname-c 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40b44c0 con 0x7f4ec413f710 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: _finish_auth 0 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: _check_auth_tickets 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ecbd4d640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:25:50.206 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4eb4001690 con 0x7f4ec413f710 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ecbd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ec8ae7640 10 monclient: dump: 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.207 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.204+0000 7f4ecbd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:50.223 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eb40017f0 con 0x7f4ec413f710 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 1 -- 172.21.15.87:0/869749216 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4eb4001c00 con 0x7f4ec413f710 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 4 set_mon_vals no callback set 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ec8ae7640 10 monclient: dump: 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.224 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.225 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:50.225 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:50.225 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:50.225 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 10 monclient: shutdown 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/869749216 >> v1:172.21.15.171:6789/0 conn(0x7f4ec413f710 legacy=0x7f4ec41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/869749216 shutdown_connections 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/869749216 >> 172.21.15.87:0/869749216 conn(0x7f4ec40b7ea0 msgr2=0x7f4ec40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/869749216 shutdown_connections 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.223+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/869749216 wait complete. 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.224+0000 7f4ecbd4d640 10 monclient: build_initial_monmap 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.224+0000 7f4ecbd4d640 10 monclient: monmap: 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:50.227 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 librados: starting msgr at 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 librados: starting objecter 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 Processor -- start 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 -- start start 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 librados: setting wanted keys 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 librados: calling monclient init 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient: init 2024-03-25T18:25:50.228 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec40a98f0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec413f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec41430c0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec4139a40 con 0x7f4ec41430c0 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec4133f20 con 0x7f4ec413f710 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40ab950 con 0x7f4ec40a98f0 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.225+0000 7f4ecbd4d640 10 monclient(hunting): authenticate will time out at 8465.520688s 2024-03-25T18:25:50.229 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.227+0000 7f4eca2ea640 1 --1- >> v1:172.21.15.87:6789/0 conn(0x7f4ec41430c0 0x7f4ec4133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6789/0 says I am v1:172.21.15.87:42386/0 (socket says 172.21.15.87:42386) 2024-03-25T18:25:50.230 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.227+0000 7f4eca2ea640 1 -- 172.21.15.87:0/61212577 learned_addr learned my addr 172.21.15.87:0/61212577 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:50.232 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.232+0000 7f4eba7fc640 10 client.?.objecter ms_handle_connect 0x7f4ec41430c0 2024-03-25T18:25:50.233 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.232+0000 7f4eba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.239+0000 7f4eba7fc640 10 client.?.objecter ms_handle_connect 0x7f4ec413f710 2024-03-25T18:25:50.240 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.239+0000 7f4eba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.243+0000 7f4eba7fc640 10 client.?.objecter ms_handle_connect 0x7f4ec40a98f0 2024-03-25T18:25:50.244 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.243+0000 7f4eba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885301659 0 0) 0x7f4ec4139a40 con 0x7f4ec41430c0 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:25:50.250 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.249+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea8004eb0 con 0x7f4ec41430c0 2024-03-25T18:25:50.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794342686 0 0) 0x7f4ec40ab950 con 0x7f4ec40a98f0 2024-03-25T18:25:50.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.262 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.261+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec4139a40 con 0x7f4ec40a98f0 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930736310 0 0) 0x7f4ec4133f20 con 0x7f4ec413f710 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:25:50.263 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.262+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40ab950 con 0x7f4ec413f710 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015722780 0 0) 0x7f4ea8004eb0 con 0x7f4ec41430c0 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ec4133f20 con 0x7f4ec41430c0 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ebc001c90 con 0x7f4ec41430c0 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.265 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.266 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.264+0000 7f4eba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395492420 0 0) 0x7f4ec40ab950 con 0x7f4ec413f710 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ea8004eb0 con 0x7f4ec413f710 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4eb4001b00 con 0x7f4ec413f710 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:50.274 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.273+0000 7f4eba7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:50.275 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.283 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 261740219 0 0) 0x7f4ec4133f20 con 0x7f4ec41430c0 2024-03-25T18:25:50.283 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 >> v1:172.21.15.87:6790/0 conn(0x7f4ec40a98f0 legacy=0x7f4ec4138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 >> v1:172.21.15.171:6789/0 conn(0x7f4ec413f710 legacy=0x7f4ec4133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 1 monclient: found mon.a 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40abc30 con 0x7f4ec41430c0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4eba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 10 monclient: _renew_subs 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec40abf10 con 0x7f4ec41430c0 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 10 monclient: _renew_subs 2024-03-25T18:25:50.284 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec4139370 con 0x7f4ec41430c0 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.282+0000 7f4ecbd4d640 1 librados: init done 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.283+0000 7f4ecbd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.283+0000 7f4ecbd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.283+0000 7f4ecbd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:50.285 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.283+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec40627b0 con 0x7f4ec41430c0 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.608+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ebc003990 con 0x7f4ec41430c0 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.608+0000 7f4eba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f4ebc003ce0 con 0x7f4ec41430c0 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4eba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4eba7fc640 10 monclient: dump: 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:50.610 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4ebbfff640 4 set_mon_vals no callback set 2024-03-25T18:25:50.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.609+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f4ebc003f00 con 0x7f4ec41430c0 2024-03-25T18:25:50.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.610+0000 7f4ec9ae9640 10 monclient: get_auth_request con 0x7f4ea806df90 auth_method 0 2024-03-25T18:25:50.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.610+0000 7f4eba7fc640 10 client.4728.objecter ms_handle_connect 0x7f4ea806df90 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f4ebc054b00 con 0x7f4ec41430c0 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 10 client.4728.objecter ms_dispatch 0x7f4ec4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ebc050610 con 0x7f4ec41430c0 2024-03-25T18:25:50.615 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:50.616 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.614+0000 7f4eba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:50.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.864+0000 7f4ecbd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.864+0000 7f4ecbd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.864+0000 7f4ecbd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:50.865 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.864+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ec40b7180 con 0x7f4ec41430c0 2024-03-25T18:25:50.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.900+0000 7f4eba7fc640 1 -- 172.21.15.87:0/61212577 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f4ebc052540 con 0x7f4ec41430c0 2024-03-25T18:25:50.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.900+0000 7f4eba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:50.901 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.900+0000 7f4eba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:50.901 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:50.901 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.902+0000 7f4ecbd4d640 10 librados: watch_flush enter 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.902+0000 7f4ecbd4d640 10 librados: watch_flush exit 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.902+0000 7f4ecbd4d640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.902+0000 7f4ecbd4d640 10 client.4728.objecter successfully canceled tick 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 >> v1:172.21.15.171:6800/3417775508 conn(0x7f4ea806df90 legacy=0x7f4ea8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 10 monclient: shutdown 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 >> v1:172.21.15.87:6789/0 conn(0x7f4ec41430c0 legacy=0x7f4ec4133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 shutdown_connections 2024-03-25T18:25:50.904 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 >> 172.21.15.87:0/61212577 conn(0x7f4ec40b7ea0 msgr2=0x7f4ec4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:50.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 shutdown_connections 2024-03-25T18:25:50.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 -- 172.21.15.87:0/61212577 wait complete. 2024-03-25T18:25:50.905 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:50.903+0000 7f4ecbd4d640 1 librados: shutdown 2024-03-25T18:25:50.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:53.494 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:41.207 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.548 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:41.262 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.590 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:11:41.303 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.598 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:41.276 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.605 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:41.284 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.618 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:41.331 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.621 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:41.334 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.631 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:11:41.310 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:53.651 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:11:41.364 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:57.722 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.835+0000 7fd7a6f04640 10 monclient: get_monmap_and_config 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.835+0000 7fd7a6f04640 10 monclient: build_initial_monmap 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.835+0000 7fd7a6f04640 10 monclient: monmap: 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:57.836 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:25:57.836245+0000 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:25:57.836245+0000 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:25:57.837 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 1 Processor -- start 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 1 -- start start 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient: init 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a00a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a013f710 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:57.838 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a01430c0 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:57.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0142520 con 0x7fd7a00a98f0 2024-03-25T18:25:57.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00b28a0 con 0x7fd7a013f710 2024-03-25T18:25:57.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00b2a30 con 0x7fd7a01430c0 2024-03-25T18:25:57.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:57.839 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.836+0000 7fd7a6f04640 10 monclient(hunting): authenticate will time out at 8473.131752s 2024-03-25T18:25:57.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.844+0000 7fd7a54a1640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7fd7a01430c0 0x7fd7a014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:57984/0 (socket says 172.21.15.87:57984) 2024-03-25T18:25:57.845 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.844+0000 7fd7a54a1640 1 -- 172.21.15.87:0/3735814162 learned_addr learned my addr 172.21.15.87:0/3735814162 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860483854 0 0) 0x7fd7a0142520 con 0x7fd7a00a98f0 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:25:57.861 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.860+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd790003520 con 0x7fd7a00a98f0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967376012 0 0) 0x7fd790003520 con 0x7fd7a00a98f0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 >> v1:172.21.15.87:6790/0 conn(0x7fd7a01430c0 legacy=0x7fd7a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 >> v1:172.21.15.171:6789/0 conn(0x7fd7a013f710 legacy=0x7fd7a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.87:6789/0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a00b44c0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd794001690 con 0x7fd7a00a98f0 2024-03-25T18:25:57.921 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd79f7fe640 10 monclient: dump: 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd7a6f04640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:25:57.922 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.919+0000 7fd7a6f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 <== mon.0 v1:172.21.15.87:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7940017f0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 1 -- 172.21.15.87:0/3735814162 <== mon.0 v1:172.21.15.87:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd794001c00 con 0x7fd7a00a98f0 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd79f7fe640 10 monclient: dump: 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:57.937 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:57.938 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd7a6f04640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:57.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd7a6f04640 4 set_mon_vals no callback set 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd7a6f04640 10 monclient: shutdown 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd7a6f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.936+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3735814162 >> v1:172.21.15.87:6789/0 conn(0x7fd7a00a98f0 legacy=0x7fd7a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.937+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3735814162 shutdown_connections 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.937+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3735814162 >> 172.21.15.87:0/3735814162 conn(0x7fd7a00b7ea0 msgr2=0x7fd7a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:57.939 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.937+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3735814162 shutdown_connections 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.937+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3735814162 wait complete. 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 10 monclient: build_initial_monmap 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 10 monclient: monmap: 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 librados: starting msgr at 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 librados: starting objecter 2024-03-25T18:25:57.940 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 Processor -- start 2024-03-25T18:25:57.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 -- start start 2024-03-25T18:25:57.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 librados: setting wanted keys 2024-03-25T18:25:57.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 1 librados: calling monclient init 2024-03-25T18:25:57.941 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.938+0000 7fd7a6f04640 10 monclient: init 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a00a98f0 addr v1:172.21.15.87:6789/0 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a013f710 addr v1:172.21.15.87:6790/0 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a01430c0 addr v1:172.21.15.171:6789/0 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0135ac0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0137030 con 0x7fd7a01430c0 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a01372a0 con 0x7fd7a013f710 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.939+0000 7fd7a6f04640 10 monclient(hunting): authenticate will time out at 8473.234448s 2024-03-25T18:25:57.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.940+0000 7fd7a54a1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd7a01430c0 0x7fd7a01e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:45276/0 (socket says 172.21.15.87:45276) 2024-03-25T18:25:57.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.940+0000 7fd7a54a1640 1 -- 172.21.15.87:0/3098666562 learned_addr learned my addr 172.21.15.87:0/3098666562 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:25:57.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.941+0000 7fd79d7fa640 10 client.?.objecter ms_handle_connect 0x7fd7a01430c0 2024-03-25T18:25:57.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.941+0000 7fd79d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:57.955 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.954+0000 7fd79d7fa640 10 client.?.objecter ms_handle_connect 0x7fd7a00a98f0 2024-03-25T18:25:57.955 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.954+0000 7fd79d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004116954 0 0) 0x7fd7a0137030 con 0x7fd7a01430c0 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.959+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784004eb0 con 0x7fd7a01430c0 2024-03-25T18:25:57.960 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157114918 0 0) 0x7fd7a0135ac0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:57.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:57.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:57.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:25:57.961 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.960+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a0137030 con 0x7fd7a00a98f0 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711522599 0 0) 0x7fd7a0137030 con 0x7fd7a00a98f0 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a0135ac0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd794001b40 con 0x7fd7a00a98f0 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 10 monclient(hunting): dump: 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.973 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.974 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.972+0000 7fd79d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:57.981 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154955153 0 0) 0x7fd784004eb0 con 0x7fd7a01430c0 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a0137030 con 0x7fd7a01430c0 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd798001b10 con 0x7fd7a01430c0 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.981+0000 7fd79d7fa640 10 monclient(hunting): dump: 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:57.982 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:57.983 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1609227568 0 0) 0x7fd7a0135ac0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 >> v1:172.21.15.87:6790/0 conn(0x7fd7a013f710 legacy=0x7fd7a01379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 >> v1:172.21.15.171:6789/0 conn(0x7fd7a01430c0 legacy=0x7fd7a01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 1 monclient: found mon.a 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0137580 con 0x7fd7a00a98f0 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:57.985 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.983+0000 7fd79d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 10 monclient: _renew_subs 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a01363b0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 10 monclient: _renew_subs 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a01353a0 con 0x7fd7a00a98f0 2024-03-25T18:25:57.986 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.984+0000 7fd7a6f04640 1 librados: init done 2024-03-25T18:25:57.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.985+0000 7fd7a6f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.985+0000 7fd7a6f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:57.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.985+0000 7fd7a6f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:57.987 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:57.985+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a00627b0 con 0x7fd7a00a98f0 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7940038a0 con 0x7fd7a00a98f0 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7fd794003bb0 con 0x7fd7a00a98f0 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 10 monclient: dump: 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.321 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79effd640 4 set_mon_vals no callback set 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.320+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7fd794019b80 con 0x7fd7a00a98f0 2024-03-25T18:25:58.322 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.321+0000 7fd79ffff640 10 monclient: get_auth_request con 0x7fd78406de30 auth_method 0 2024-03-25T18:25:58.323 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.321+0000 7fd79d7fa640 10 client.4734.objecter ms_handle_connect 0x7fd78406de30 2024-03-25T18:25:58.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7fd794053720 con 0x7fd7a00a98f0 2024-03-25T18:25:58.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 10 client.4734.objecter ms_dispatch 0x7fd7a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:58.328 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 3 client.4734.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:58.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 3 client.4734.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:58.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T18:25:58.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd79404f060 con 0x7fd7a00a98f0 2024-03-25T18:25:58.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:58.329 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.327+0000 7fd79d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.574+0000 7fd7a6f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.574+0000 7fd7a6f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.574+0000 7fd7a6f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.87:6789/0 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.574+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 --> v1:172.21.15.87:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a0074cc0 con 0x7fd7a00a98f0 2024-03-25T18:25:58.618 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.617+0000 7fd79d7fa640 1 -- 172.21.15.87:0/3098666562 <== mon.0 v1:172.21.15.87:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7fd794051160 con 0x7fd7a00a98f0 2024-03-25T18:25:58.618 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.617+0000 7fd79d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:58.618 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.617+0000 7fd79d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:58.619 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:25:58.619 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 10 librados: watch_flush enter 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 10 librados: watch_flush exit 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 10 client.4734.objecter successfully canceled tick 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 >> v1:172.21.15.171:6800/3417775508 conn(0x7fd78406de30 legacy=0x7fd7840702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 10 monclient: shutdown 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 >> v1:172.21.15.87:6789/0 conn(0x7fd7a00a98f0 legacy=0x7fd7a0133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.619+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 shutdown_connections 2024-03-25T18:25:58.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.620+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 >> 172.21.15.87:0/3098666562 conn(0x7fd7a00b7ea0 msgr2=0x7fd7a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:58.622 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.620+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 shutdown_connections 2024-03-25T18:25:58.622 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.620+0000 7fd7a6f04640 1 -- 172.21.15.87:0/3098666562 wait complete. 2024-03-25T18:25:58.622 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:25:58.620+0000 7fd7a6f04640 1 librados: shutdown 2024-03-25T18:25:58.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:05.441 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.556+0000 7f2dfd28f640 10 monclient: get_monmap_and_config 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.556+0000 7f2dfd28f640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.556+0000 7f2dfd28f640 10 monclient: monmap: 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:26:05.557051+0000 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:26:05.557051+0000 2024-03-25T18:26:05.557 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.558 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.558 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:26:05.558 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-03-25T18:26:05.558 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-b 2024-03-25T18:26:05.558 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.556+0000 7f2dfd28f640 1 Processor -- start 2024-03-25T18:26:05.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 1 -- start start 2024-03-25T18:26:05.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient: init 2024-03-25T18:26:05.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.559 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-a con 0x7f2df80705b0 addr v1:172.21.15.87:6789/0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-c con 0x7f2df81402d0 addr v1:172.21.15.171:6789/0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-b con 0x7f2df8149ae0 addr v1:172.21.15.87:6790/0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80718c0 con 0x7f2df80705b0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80714c0 con 0x7f2df81402d0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df8072b50 con 0x7f2df8149ae0 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.560 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.557+0000 7f2dfd28f640 10 monclient(hunting): authenticate will time out at 8480.852646s 2024-03-25T18:26:05.565 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.565+0000 7f2df67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2df81402d0 0x7f2df8140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.87:49940/0 (socket says 172.21.15.87:49940) 2024-03-25T18:26:05.566 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.565+0000 7f2df67fc640 1 -- 172.21.15.87:0/1543614748 learned_addr learned my addr 172.21.15.87:0/1543614748 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:26:05.578 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941068724 0 0) 0x7f2df80714c0 con 0x7f2df81402d0 2024-03-25T18:26:05.578 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.578 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.578 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.578 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:26:05.579 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.577+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de0003520 con 0x7f2df81402d0 2024-03-25T18:26:05.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366691677 0 0) 0x7f2df8072b50 con 0x7f2df8149ae0 2024-03-25T18:26:05.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.586 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.585+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df80714c0 con 0x7f2df8149ae0 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816196509 0 0) 0x7f2df80718c0 con 0x7f2df80705b0 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:26:05.587 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.586+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8072b50 con 0x7f2df80705b0 2024-03-25T18:26:05.601 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131134554 0 0) 0x7f2de0003520 con 0x7f2df81402d0 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 >> v1:172.21.15.87:6790/0 conn(0x7f2df8149ae0 legacy=0x7f2df814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 >> v1:172.21.15.87:6789/0 conn(0x7f2df80705b0 legacy=0x7f2df813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df813dd10 con 0x7f2df81402d0 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.602 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2dec001690 con 0x7f2df81402d0 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2df5ffb640 10 monclient: dump: 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:26:05.603 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:26:05.604 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2dfd28f640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:26:05.604 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.598+0000 7f2dfd28f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:05.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dec0017f0 con 0x7f2df81402d0 2024-03-25T18:26:05.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 1 -- 172.21.15.87:0/1543614748 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2dec001c00 con 0x7f2df81402d0 2024-03-25T18:26:05.607 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2df5ffb640 10 monclient: dump: 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.608 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2dfd28f640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:05.609 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.606+0000 7f2dfd28f640 4 set_mon_vals no callback set 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 10 monclient: shutdown 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 1 -- 172.21.15.87:0/1543614748 >> v1:172.21.15.171:6789/0 conn(0x7f2df81402d0 legacy=0x7f2df8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 1 -- 172.21.15.87:0/1543614748 shutdown_connections 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 1 -- 172.21.15.87:0/1543614748 >> 172.21.15.87:0/1543614748 conn(0x7f2df80b5d10 msgr2=0x7f2df80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 1 -- 172.21.15.87:0/1543614748 shutdown_connections 2024-03-25T18:26:05.611 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.607+0000 7f2dfd28f640 1 -- 172.21.15.87:0/1543614748 wait complete. 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.608+0000 7f2dfd28f640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.608+0000 7f2dfd28f640 10 monclient: monmap: 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:epoch 0 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:last_changed 0.000000 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:created 0.000000 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 1 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.noname-a 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.noname-c 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.608+0000 7f2dfd28f640 1 librados: starting msgr at 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.608+0000 7f2dfd28f640 1 librados: starting objecter 2024-03-25T18:26:05.612 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 1 Processor -- start 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 1 -- start start 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 1 librados: setting wanted keys 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 1 librados: calling monclient init 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 10 monclient: init 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.609+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-c con 0x7f2df80705b0 addr v1:172.21.15.87:6790/0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-a con 0x7f2df81402d0 addr v1:172.21.15.87:6789/0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 10 monclient(hunting): picked mon.noname-b con 0x7f2df8149ae0 addr v1:172.21.15.171:6789/0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 1 -- --> v1:172.21.15.87:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df8137fb0 con 0x7f2df81402d0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df81d80d0 con 0x7f2df8149ae0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 1 -- --> v1:172.21.15.87:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df8137a60 con 0x7f2df80705b0 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.613 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.610+0000 7f2dfd28f640 10 monclient(hunting): authenticate will time out at 8480.905012s 2024-03-25T18:26:05.618 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.618+0000 7f2df6ffd640 1 --1- >> v1:172.21.15.87:6790/0 conn(0x7f2df80705b0 0x7f2df806d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.87:6790/0 says I am v1:172.21.15.87:34630/0 (socket says 172.21.15.87:34630) 2024-03-25T18:26:05.619 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.618+0000 7f2df6ffd640 1 -- 172.21.15.87:0/2688520046 learned_addr learned my addr 172.21.15.87:0/2688520046 (peer_addr_for_me v1:172.21.15.87:0/0) 2024-03-25T18:26:05.619 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.618+0000 7f2ddf7fe640 10 client.?.objecter ms_handle_connect 0x7f2df80705b0 2024-03-25T18:26:05.619 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.618+0000 7f2ddf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.620+0000 7f2ddf7fe640 10 client.?.objecter ms_handle_connect 0x7f2df8149ae0 2024-03-25T18:26:05.621 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.620+0000 7f2ddf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648437639 0 0) 0x7f2df81d80d0 con 0x7f2df8149ae0 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:26:05.627 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.626+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de4004eb0 con 0x7f2df8149ae0 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 client.?.objecter ms_handle_connect 0x7f2df81402d0 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.2 v1:172.21.15.87:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364923488 0 0) 0x7f2df8137a60 con 0x7f2df80705b0 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:26:05.633 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.632+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.87:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df81d80d0 con 0x7f2df80705b0 2024-03-25T18:26:05.648 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.0 v1:172.21.15.87:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117009378 0 0) 0x7f2df813dd10 con 0x7f2df81402d0 2024-03-25T18:26:05.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:26:05.649 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.648+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.87:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8137a60 con 0x7f2df81402d0 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129115515 0 0) 0x7f2de4004eb0 con 0x7f2df8149ae0 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2df813dd10 con 0x7f2df8149ae0 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2df0001bd0 con 0x7f2df8149ae0 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.650 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.651 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.649+0000 7f2ddf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:05.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.659+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.2 v1:172.21.15.87:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4153893039 0 0) 0x7f2df81d80d0 con 0x7f2df80705b0 2024-03-25T18:26:05.660 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.659+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.87:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2de4004eb0 con 0x7f2df80705b0 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.660+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.2 v1:172.21.15.87:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2de80019f0 con 0x7f2df80705b0 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.660+0000 7f2ddf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.660+0000 7f2ddf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.660+0000 7f2ddf7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:26:05.661 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:26:05.662 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1246778534 0 0) 0x7f2df813dd10 con 0x7f2df8149ae0 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 >> v1:172.21.15.87:6790/0 conn(0x7f2df80705b0 legacy=0x7f2df806d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 >> v1:172.21.15.87:6789/0 conn(0x7f2df81402d0 legacy=0x7f2df81daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 1 monclient: found mon.b 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8137bf0 con 0x7f2df8149ae0 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.663 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2ddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 10 monclient: _renew_subs 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df8148ee0 con 0x7f2df8149ae0 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 10 monclient: _renew_subs 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df806d930 con 0x7f2df8149ae0 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.661+0000 7f2dfd28f640 1 librados: init done 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.662+0000 7f2dfd28f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.662+0000 7f2dfd28f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.662+0000 7f2dfd28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:26:05.664 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.662+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df81d8260 con 0x7f2df8149ae0 2024-03-25T18:26:05.936 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2df0003970 con 0x7f2df8149ae0 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3772618545 0 0) 0x7f2df0002100 con 0x7f2df8149ae0 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2ddf7fe640 10 monclient: dump: 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:epoch 1 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:fsid 97b2a0dc-69f2-4b94-95e9-3722e689d10f 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:last_changed 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:created 2024-03-25T18:13:39.397301+0000 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:election_strategy: 3 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:0: v1:172.21.15.87:6789/0 mon.a 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-03-25T18:26:05.937 INFO:teuthology.misc.health.smithi087.stderr:2: v1:172.21.15.87:6790/0 mon.c 2024-03-25T18:26:05.938 INFO:teuthology.misc.health.smithi087.stderr: 2024-03-25T18:26:05.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.935+0000 7f2df57fa640 4 set_mon_vals no callback set 2024-03-25T18:26:05.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.936+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2664551696 0 0) 0x7f2df0019bb0 con 0x7f2df8149ae0 2024-03-25T18:26:05.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.937+0000 7f2df6ffd640 10 monclient: get_auth_request con 0x7f2de406df70 auth_method 0 2024-03-25T18:26:05.938 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.937+0000 7f2ddf7fe640 10 client.4738.objecter ms_handle_connect 0x7f2de406df70 2024-03-25T18:26:05.942 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.941+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2532940291 0 0) 0x7f2df00537d0 con 0x7f2df8149ae0 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 10 client.4738.objecter ms_dispatch 0x7f2df8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2df004f2e0 con 0x7f2df8149ae0 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.943 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:05.942+0000 7f2ddf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:06.193 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.192+0000 7f2dfd28f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.193 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.192+0000 7f2dfd28f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.194 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.192+0000 7f2dfd28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-03-25T18:26:06.194 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.192+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2df80627b0 con 0x7f2df8149ae0 2024-03-25T18:26:06.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.215+0000 7f2ddf7fe640 1 -- 172.21.15.87:0/2688520046 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3520440095) 0x7f2df0051210 con 0x7f2df8149ae0 2024-03-25T18:26:06.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.215+0000 7f2ddf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.216 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.215+0000 7f2ddf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:06.216 INFO:teuthology.misc.health.smithi087.stdout: 2024-03-25T18:26:06.216 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"97b2a0dc-69f2-4b94-95e9-3722e689d10f","created":"2024-03-25T18:14:39.395261+0000","modified":"2024-03-25T18:24:43.223875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:43.223875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"980ca77f-5be5-4342-81e2-07ef296105f3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"db380654-84f4-4fd2-b2a0-d94129e518b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c31c9f22-b8b1-4060-bbfc-18fdf310a080","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"18f5b86f-145b-4d76-b45d-f921a4b316df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a6cae3ee-630b-444f-84bd-bd172830fcab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a7288f1-5ccf-4d22-92fc-69a1d3521ccc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89fa7866-1bf5-408f-924d-0d7fba1d5565","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"526fd20b-e3d4-43e6-b854-52268cb06e06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 10 librados: watch_flush enter 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 10 librados: watch_flush exit 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 >> v1:172.21.15.171:6800/3417775508 conn(0x7f2de406df70 legacy=0x7f2de4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 10 monclient: shutdown 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 >> v1:172.21.15.171:6789/0 conn(0x7f2df8149ae0 legacy=0x7f2df81db140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 shutdown_connections 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 >> 172.21.15.87:0/2688520046 conn(0x7f2df80b5d10 msgr2=0x7f2df80b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 shutdown_connections 2024-03-25T18:26:06.219 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.217+0000 7f2dfd28f640 1 -- 172.21.15.87:0/2688520046 wait complete. 2024-03-25T18:26:06.220 INFO:teuthology.misc.health.smithi087.stderr:2024-03-25T18:26:06.218+0000 7f2dfd28f640 1 librados: shutdown 2024-03-25T18:26:06.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:06.230 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:26:06.232 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:26:06.232 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:26:06.232 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:08.135 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:11:23.318 35205== 2024-03-25T18:26:08.136 INFO:tasks.ceph.osd.0.smithi087.stderr:==00:00:11:23.318 35205== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:08.172 INFO:tasks.ceph.osd.0.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:12.335 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:26:12.336 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:26:12.336 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:26:12.336 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:13.573 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:01.286 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.616 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:01.294 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.619 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:01.297 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.637 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:01.351 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.643 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:01.321 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.670 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:01.383 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.701 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:01.414 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.705 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:01.418 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.732 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:01.445 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:14.201 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:11:29.399 35204== 2024-03-25T18:26:14.201 INFO:tasks.ceph.osd.1.smithi087.stderr:==00:00:11:29.399 35204== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:14.236 INFO:tasks.ceph.osd.1.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:18.438 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:26:18.438 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:26:18.439 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:26:18.439 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:20.453 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:11:35.633 35206== 2024-03-25T18:26:20.453 INFO:tasks.ceph.osd.2.smithi087.stderr:==00:00:11:35.633 35206== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:20.494 INFO:tasks.ceph.osd.2.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:24.541 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:26:24.542 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:26:24.542 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:26:24.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:26.395 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:11:41.574 35207== 2024-03-25T18:26:26.395 INFO:tasks.ceph.osd.3.smithi087.stderr:==00:00:11:41.574 35207== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:26.442 INFO:tasks.ceph.osd.3.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:30.648 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:26:30.648 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:26:30.649 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:26:30.649 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:32.734 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:11:47.928 35348== 2024-03-25T18:26:32.734 INFO:tasks.ceph.osd.4.smithi171.stderr:==00:00:11:47.928 35348== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:32.769 INFO:tasks.ceph.osd.4.smithi171.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:33.640 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:21.353 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.646 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:21.360 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.672 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:21.385 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.698 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:21.377 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.699 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:21.412 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.702 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:21.381 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.704 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:21.417 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.717 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:21.396 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.718 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:21.431 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:36.751 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:26:36.751 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:26:36.752 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:26:36.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:38.586 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:11:53.783 35347== 2024-03-25T18:26:38.587 INFO:tasks.ceph.osd.5.smithi171.stderr:==00:00:11:53.783 35347== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:38.624 INFO:tasks.ceph.osd.5.smithi171.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:42.854 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:26:42.854 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:26:42.855 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:26:42.855 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:44.668 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:11:59.845 35349== 2024-03-25T18:26:44.668 INFO:tasks.ceph.osd.6.smithi171.stderr:==00:00:11:59.845 35349== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:44.715 INFO:tasks.ceph.osd.6.smithi171.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:48.957 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:26:48.957 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:26:48.957 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:26:48.958 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:50.514 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:12:05.688 35350== 2024-03-25T18:26:50.515 INFO:tasks.ceph.osd.7.smithi171.stderr:==00:00:12:05.688 35350== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:50.545 INFO:tasks.ceph.osd.7.smithi171.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:53.691 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:41.404 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.693 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:41.407 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.705 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:41.418 34893== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.737 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:41.450 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.746 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:41.424 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.747 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:41.426 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.752 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:41.465 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.758 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:12:41.437 34802== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.766 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:41.479 34894== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:55.059 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi171 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:26:55.060 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:26:55.060 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:26:55.060 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:26:55.060 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:55.089 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:26:55.090 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:26:55.090 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:55.091 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:26:55.090+0000 7fd385922640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34800) UID: 0 2024-03-25T18:26:55.091 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-03-25T18:26:55.090+0000 7fd385922640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:26:55.119 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:26:55.120 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:26:55.120 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:26:55.120 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:55.149 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-03-25T18:26:55.133+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34859) UID: 0 2024-03-25T18:26:55.150 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-03-25T18:26:55.135+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:26:55.554 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:43.267 34893== 2024-03-25T18:26:55.554 INFO:tasks.ceph.mon.a.smithi087.stderr:==00:00:12:43.267 34893== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:55.563 INFO:tasks.ceph.mon.a.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:27:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:27:01.222 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:27:01.222 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:27:01.223 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:27:01.223 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:01.238 INFO:tasks.ceph.mon.c.smithi087.stderr:2024-03-25T18:27:01.225+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34861) UID: 0 2024-03-25T18:27:01.239 INFO:tasks.ceph.mon.c.smithi087.stderr:2024-03-25T18:27:01.227+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:27:01.906 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:49.618 34894== 2024-03-25T18:27:01.906 INFO:tasks.ceph.mon.c.smithi087.stderr:==00:00:12:49.618 34894== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:27:01.914 INFO:tasks.ceph.mon.c.smithi087.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:27:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:27:07.324 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:27:07.324 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:27:07.325 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:27:07.325 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:07.341 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-03-25T18:27:07.328+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34798) UID: 0 2024-03-25T18:27:07.341 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-03-25T18:27:07.330+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:32:06.398 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:32:06.399 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:32:06.454 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:32:06.454 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:32:06.528 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:32:06.528 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:32:06.578 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:32:06.579 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:32:06.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:32:06.635 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:32:06.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:32:06.694 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:32:06.834 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:32:06.834 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:32:06.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:32:06.891 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:32:06.942 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:32:06.943 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:32:06.991 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:32:06.994 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/data/mon.a.tgz 2024-03-25T18:32:06.994 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-03-25T18:32:07.009 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.joAkCgKnMT 2024-03-25T18:32:07.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.joAkCgKnMT 2024-03-25T18:32:07.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.joAkCgKnMT 2024-03-25T18:32:07.193 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.joAkCgKnMT is 133KB 2024-03-25T18:32:07.245 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.joAkCgKnMT 2024-03-25T18:32:07.261 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/data/mon.c.tgz 2024-03-25T18:32:07.262 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-03-25T18:32:07.312 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.xEsPaUJWuY 2024-03-25T18:32:07.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xEsPaUJWuY 2024-03-25T18:32:07.400 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.xEsPaUJWuY 2024-03-25T18:32:07.483 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.xEsPaUJWuY is 133KB 2024-03-25T18:32:07.534 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.xEsPaUJWuY 2024-03-25T18:32:07.548 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/data/mon.b.tgz 2024-03-25T18:32:07.548 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-03-25T18:32:07.562 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.vT2YKWHYYh 2024-03-25T18:32:07.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vT2YKWHYYh 2024-03-25T18:32:07.658 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.vT2YKWHYYh 2024-03-25T18:32:07.742 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.vT2YKWHYYh is 133KB 2024-03-25T18:32:07.764 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.vT2YKWHYYh 2024-03-25T18:32:07.778 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:32:07.778 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:32:07.780 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:32:08.107 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:32:08.179 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T18:32:08.368 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:32:08.371 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi087/crash 2024-03-25T18:32:08.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:32:08.399 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi171/crash 2024-03-25T18:32:08.400 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:32:08.426 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:32:08.426 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:32:08.444 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35117.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35117.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35117.log.gz 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:32:08.514 INFO:teuthology.orchestra.run.smithi171.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr: 69.7% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34957.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/valgrind/mon.b.log: /var/log/ceph/valgrind/osd.4.log: 55.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35197.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35157.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35077.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.34957.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35197.log: /var/log/ceph/ceph-client.admin.35157.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35077.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35197.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35157.log.gz 2024-03-25T18:32:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35077.log.gz 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35037.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35248.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35037.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.4.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35037.log.gz -5 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34997.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-03-25T18:32:08.516 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:32:08.522 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-03-25T18:32:08.522 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-03-25T18:32:08.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:32:08.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-03-25T18:32:08.522 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41165.log: /var/log/ceph/ceph-client.admin.39169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40224.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39052.log 87.0% 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38776.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39052.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz -5 --verbose 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.41789.log 2024-03-25T18:32:08.523 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39052.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41243.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38776.log: /var/log/ceph/ceph-client.admin.41789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.41243.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41009.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41243.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41009.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/mon.c.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41009.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 72.0%/var/log/ceph/valgrind/osd.0.log: -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:32:08.524 INFO:teuthology.orchestra.run.smithi087.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40107.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34901.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41907.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40107.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41867.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.34901.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41867.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41907.log.gz 2024-03-25T18:32:08.525 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41867.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39561.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40578.log: gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.41360.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39091.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35230.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35203.log 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-03-25T18:32:08.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39091.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35203.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.35230.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38818.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35203.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39404.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38818.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39013.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40500.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39404.log.gz 2024-03-25T18:32:08.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39013.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39834.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39013.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39834.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39600.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-03-25T18:32:08.528 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38935.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39326.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42024.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39326.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.42024.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39326.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42024.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-03-25T18:32:08.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38896.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39444.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37142.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.37142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39365.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38896.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37142.log.gz 2024-03-25T18:32:08.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.35348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39247.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39912.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39951.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38857.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39951.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39912.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29036.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39756.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38857.log: /var/log/ceph/ceph.tmp-client.admin.29036.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29036.log.gz 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41946.log 2024-03-25T18:32:08.531 INFO:teuthology.orchestra.run.smithi087.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38857.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.39756.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40146.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40539.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38974.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41438.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40539.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41750.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40539.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41477.log 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41438.log.gz 2024-03-25T18:32:08.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41477.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41750.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.40344.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40656.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41477.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41126.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40344.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41126.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41126.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.41282.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40461.log 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41282.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-03-25T18:32:08.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36179.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40461.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41594.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40461.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.36179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41516.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9%/var/log/ceph/ceph-client.admin.39483.log: -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36179.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41828.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41516.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41828.log: -- /var/log/ceph/ceph-client.admin.39678.log 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41828.log.gz 2024-03-25T18:32:08.534 INFO:teuthology.orchestra.run.smithi087.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41321.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41321.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41321.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41672.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz/var/log/ceph/ceph-client.admin.40617.log: 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.40814.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.41672.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41672.log.gz 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39639.log 2024-03-25T18:32:08.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.39639.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gz 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39795.log: -- /var/log/ceph/ceph-client.admin.40853.log 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39639.log.gz 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38737.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-03-25T18:32:08.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.40853.log: /var/log/ceph/ceph-mon.c.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40853.log.gz 2024-03-25T18:32:08.580 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:32:09.247 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:17:56.925 34802== 2024-03-25T18:32:09.247 INFO:tasks.ceph.mon.b.smithi171.stderr:==00:00:17:56.925 34802== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:32:09.263 INFO:tasks.ceph.mon.b.smithi171.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:32:09.299 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:32:09.300 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:32:09.409 INFO:teuthology.orchestra.run.smithi171.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:32:09.521 INFO:teuthology.orchestra.run.smithi171.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:32:09.541 INFO:teuthology.orchestra.run.smithi171.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:32:09.597 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:32:09.599 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:32:09.680 INFO:teuthology.orchestra.run.smithi171.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:32:13.610 INFO:teuthology.orchestra.run.smithi087.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:32:13.693 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T18:32:13.749 INFO:teuthology.orchestra.run.smithi171.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:32:13.751 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-03-25T18:32:13.751 INFO:teuthology.orchestra.run.smithi171.stderr:real 0m5.263s 2024-03-25T18:32:13.751 INFO:teuthology.orchestra.run.smithi171.stderr:user 0m9.125s 2024-03-25T18:32:13.751 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m0.355s 2024-03-25T18:32:13.826 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:32:13.829 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-25T18:32:13.829 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m5.373s 2024-03-25T18:32:13.829 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m13.920s 2024-03-25T18:32:13.829 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.494s 2024-03-25T18:32:13.829 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:32:13.829 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi087/log 2024-03-25T18:32:13.830 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:32:16.293 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi171/log 2024-03-25T18:32:16.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:32:17.759 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:32:17.970 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=879d820ea2884444beaf50920ccaa39b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:32:17.977 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:32:17.986 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:32:17.996 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:32:17.996 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:32:18.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:32:18.084 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:32:18.093 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:32:18.093 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:32:18.096 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:32:18.109 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:32:18.111 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:32:18.113 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:32:18.113 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-03-25T18:32:18.113 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +683us[ +683us] +/- 71ms 2024-03-25T18:32:18.114 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -1085us[-1035us] +/- 56ms 2024-03-25T18:32:18.114 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +848us[ +898us] +/- 76ms 2024-03-25T18:32:18.114 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +657us[ +657us] +/- 71ms 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -1109us[-1105us] +/- 56ms 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +819us[ +823us] +/- 76ms 2024-03-25T18:32:18.116 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:32:18.117 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:32:18.128 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:32:18.129 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:32:18.139 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:32:18.225 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:32:18.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:32:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:18.327 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:32:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:18.358 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:32:18.367 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:32:18.377 INFO:teuthology.task.internal:Duration was 8517.262419 seconds 2024-03-25T18:32:18.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:32:18.386 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:32:18.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:32:18.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:32:18.431 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:32:18.442 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:32:18.886 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:32:18.886 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-03-25T18:32:18.887 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:32:18.907 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-03-25T18:32:18.907 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:32:18.927 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:32:18.927 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:32:18.949 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:32:18.996 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:32:18.996 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:32:19.019 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:32:19.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:32:19.151 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:32:19.152 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:32:19.178 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:32:19.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:32:19.214 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:19.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:19.243 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-03-25T18:32:19.271 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-03-25T18:32:19.304 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:19.323 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:19.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:32:19.368 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:32:19.369 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi087 2024-03-25T18:32:19.369 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:32:19.405 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247/remote/smithi171 2024-03-25T18:32:19.405 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:32:19.440 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:32:19.440 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:32:19.442 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:32:19.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:32:19.503 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:32:19.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:32:19.512 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:32:19.512 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:32:19.514 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:32:19.526 INFO:teuthology.orchestra.run.smithi087.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:32 /home/ubuntu/cephtest 2024-03-25T18:32:19.547 INFO:teuthology.orchestra.run.smithi171.stdout: 265835 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:32 /home/ubuntu/cephtest 2024-03-25T18:32:19.548 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:32:19.558 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:32:19.577 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:32:19.598 DEBUG:teuthology.nuke:shortname: smithi087 2024-03-25T18:32:19.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:32:19.622 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:01:51.987156', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIgV9nVxjA/tkjSmI2/rMBJUsU3ZgOjTL1q0uxeQIB2jJz7E19yBi0JwhUL3MOe4vvRZsUDnne1+evrxO55qos0='} 2024-03-25T18:32:19.623 DEBUG:teuthology.nuke:shortname: smithi171 2024-03-25T18:32:19.623 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:32:19.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621247', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:01:51.989150', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBt6g2fh+YzZnvAAW6vAoxLl/uWoK/Ld3GV5Mvn9yC9qpXLYU65zgeGJ1QY6FY0NWTToyw6y3UDkkKKlSlz75Ec='} 2024-03-25T18:32:19.673 INFO:teuthology.orchestra.console.smithi087:Power off 2024-03-25T18:32:19.674 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:32:19.696 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2024-03-25T18:32:19.697 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:19.710 INFO:teuthology.orchestra.console.smithi171:Power off 2024-03-25T18:32:19.711 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:32:19.719 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-03-25T18:32:19.731 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-03-25T18:32:19.732 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:19.755 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-25T18:32:23.720 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:23.756 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:23.831 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-03-25T18:32:23.865 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-03-25T18:32:27.834 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:27.865 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:27.944 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-03-25T18:32:27.976 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-03-25T18:32:28.045 INFO:teuthology.orchestra.console.smithi087:Power off completed 2024-03-25T18:32:28.077 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-03-25T18:32:28.182 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 8517.26241850853 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=879d820ea2884444beaf50920ccaa39b status: fail success: false 2024-03-25T18:32:28.183 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:32:28.264 INFO:teuthology.run:FAIL